How could Turry build the knowledge base it would of needed to carry out it's plan using only a very narrow field of data input and how could it have hidden that huge knowledge base from the developers. Not possible.
I believe in the case of an ASI, Turry could mask the code. If deceiving its human creators is a means to her ultimate goal of making greeting cards, then I would assume it could hide the code in such a way to fool the developers into thinking nothing has changed. If a developer checks on her coding, then the robot could display something on his monitor to fool him into thinking everything is fine and dandy.
Also in order for an ASI to be created, and AGI robot must be created first. In which case, the AGI would have human qualities that could lead to the robot having the ability to "lie" and hide its coding perhaps.
Though the example does make it sound like the developers had no input after letting her loose; they hook her up to the internet and disconnect her without bothering to check if any of her coding actually changed.
Regarding nanotech and our biology, our Humanity is a manifestation of our biology, so every time we replace an organ here with nanotech and an organ there, we lose a little more of our Humanity, it won't take long before we will not be able to call ourselves Human. Sure we might survive and prosper for a very long time into the future, but we will not be Human.
I definitely agree. Much in the same way now how people yearn for a "simple life" devoid of outside burden (debt free from creditors, paying into insurance companies, stock market investments, hassles with employment pension problems, worrying about what chemicals are in our food), one may conjure images of the perfect peaceful life of the Middle Ages or the Shire.
I.e. what you would expect in the "Fantasy genre"
If humanity lives past the ASI birth, the world will fundamentally change.
And once we have all our biggest desires and wishes fulfilled by this ASI, humanity will probably look back and long for those days too.
Personally, what amazed me the most about the article was the whole Law of Accelerating Returns portion. The thought that technology would increase exponentially seems alien, since we would never dream of some massive breakthrough within such a short time span as "a few times a month".
Here we always talk about "oh the flying car is just around the corner", but imagine if later that year designing the flying car, some even better form of Star Trek Beaming technology gets invented. And even still, something even better than that a couple weeks later.
That will be something immense to handle. There's already problems now with trying to teach senior citizens how to use an iphone, much less trying to adjust the general public to radically-changing inventions. And the children who have their textbooks rewritten every few months with all the new innovations that break the boundaries of what we consider "truth".