Socratic Monologue

I have misgivings. Not so much about creating intelligent beings where none were before (new intellegent beings are born every minute), but moving too quickly on tech that we don't sufficiently understand (and possibly cannot ever sufficiently understand). Even internal combustion engines have become a difficult to control monster, and that happened over many decades. Restricting a bro's coal rolling lifted truck is somewhat challenging currently; imagine trying to take away his "assistant".

Aside from the political sorts of concerns, we simply don't (and I'd argue 'can't' because consciousness is non-physical and subjective) understand what makes an intelligent system sentient/conscious. As of about ten years ago anyway, the best explanation philosophers of mind had is that sentience is an "emergent" property of any system that's complex enough, whether that system is made of neurons or silicon chips or lincoln logs. That means that when a AI researcher says 'this AI isn't actually conscious' they don't know that, since we don't know at what level of complexity consciousness emerges. It is arguably at a level far below the complexity of the human brain, since many fairly simply animals -- which are possibly less "intelligent" than current AI -- behave in ways consistent with consciousness.

Here's a great summary of the state of our knowledge (yes, most of that should look pretty incomprehensible, because it is).