Granted, it is more fun to have more answers involved, but 2 identical answers immediately gives it away as fake.

You do it wrong, you provided the “answer” to the logic proposition, and got a parroted the proof for it.

Well, that’s the same situation I was in and just what I did. For that matter, Peano was also in that situation.

This is fixed now, and had to do with tokenizing info incorrectly.

Not quite. It’s a fundamental part of tokenization. The LLM does not “see” the individual letters. By, for example, adding spaces between the letters one could force a different tokenization and a correct count (I tried back then). It’s interesting that the LLM counted 2 "r"s, as that is phonetically correct. One wonders how it picks up on these things. It’s not really clear why it should be able to count at all.

It’s possible to make an LLM work on individual letters, but that is computationally inefficient. A few months ago, researchers at Meta proposed a possible solution called the Byte Latent Transformer (BLT). We’ll see if anything comes of it.

In any case, I do not see the relation to consciousness. Certainly there are enough people who are not able to spell or count and one would not say that they lack consciousness, I assume.

Yes, but if you instruct a parrot or LLM to say yes when asked if it is separate from it’s surroundings, it doesn’t mean it is just because it says so.

That’s true. We need to observe the LLM in its natural habit. What an LLM typically does, is continue a text. (It could also be used to work backwards or fill in the middle, but never mind.) A base model is no good as a chatbot. It has to be instruct-tuned. In operation, the tuned model is given a chat log containing a system prompt, text from the user, and text that it has previously generated. It will then add a reply and terminate the output. This text, the chat log, could be said to be the sum of its “sensory perceptions” as well as its “short-term memory”. Within this, it is able to distinguish its own replies, that of the user, and possibly other texts.

My example shows this level of understanding clearly isn’t there.

Can you lay out what abilities are connected to consciousness? What tasks are diagnostic of consciousness? Could we use an IQ test and diagnose people as having or not consciousness?

I was a bit confused by that question, because consciousness is not a construct, the brain is, of which consciousness is an emerging property.

The brain is a physical object. Consciousness is both an emergent property and a construct; like, say, temperature or IQ.

You are saying that there are different levels of consciousness. So, it must be something that is measurable and quantifiable. I assume a consciousness test would be similar to IQ test in that it would contain selected “puzzles”.

We have to figure out how consciousness is different from IQ. What puzzles are diagnostic of consciousness and not of academic ability?

It’s annoying that 25% appears twice. How about these answers:

a) 100%

b) 75%

c) 50%

d) 0%

Because I don’t think we have a sure methodology.

I don’t think there’s an agreed definition.

Strong AI or AGI, or whatever you will, is usually talked about in terms of intellectual ability. It’s not quite clear why this would require consciousness. Some tasks are aided by or maybe even necessitate self-awareness; for example, chatbots. But it seems to me that you could leave out such tasks and still have something quite impressive.

Then, of course, there is no agreed definition of consciousness. Many will argue that the self-awareness of chatbots is not consciousness.

I would say most people take strong AI and similar to mean an artificial person, for which they take consciousness as a necessary ingredient. Of course, it is impossible to engineer an artificial person. It is like creating a technology to turn a peasant into a king. It is a category error. A less kind take could be that stochastic parrots string words together based on superficial patterns without any understanding.

But we may be able to prove that it is NOT conscious, which I think is clearly the case with current level AI. Although you don’t accept the example I provided, I believe it is clear evidence of lack of a consciousness behind the high level of intelligence it clearly has.

Indeed, I do not see the relation between consciousness and reasoning in this example.

Self-awareness means the ability to distinguish self from other, which implies computing from sensory data what is oneself and what is not. That could be said to be a form of reasoning. But I do not see such a relation for the example.

By that standard, are all humans conscious?

FWIW, I asked GPT-4o mini via DDG.

I don’t know if that means it understands. It’s how I would have done it (yesterday, after looking up Peano Axioms in Wikipedia), and I don’t know if I understand it.

Just because you can’t make a mathematical proof doesn’t mean you don’t understand the very simple truth of the statement.

If I can’t prove it, I don’t know how I can claim to understand it.

It’s axiomatic that equality is symmetric. It’s also axiomatic that 1+1=2. There is not a whole lot to understand. I have memorized that. Actually, having now thought about this for a bit, I think I can prove it.

What makes the difference between a human learning these things and an AI being trained for them?

I think if I could describe that, I might actually have solved the problem of strong AI.

Then how will you know the difference between strong AI and not-strong AI?

I don’t see why the example requiring training for humans to understand is unfortunate.

Humans aren’t innately good at math. I wouldn’t have been able to prove the statement without looking things up. I certainly would not be able to come up with the Peano Axioms, or anything comparable, on my own. Most people, even educated people, probably wouldn’t understand what there is to prove. Actually, I’m not sure if I do.

It’s not clear why such deficiencies among humans do not argue against human consciousness.

A leading AI has way more training than would ever be possible for any human, still they don’t grasp basic concepts, while their knowledge is way bigger than for any human.

That’s dubious. LLMs are trained on more text than a human ever sees, but humans are trained on data from several senses. I guess it’s not entirely clear how much data that is, but it’s a lot and very high quality. Humans are trained on that sense data and not on text. Humans read text and may learn from it.

Being conscious is not just to know what the words mean, but to understand what they mean.

What might an operational definition look like?

Obviously the Turing test doesn’t cut it, which I suspected already back then.

The Turing test is misunderstood a lot. Here’s Wikipedia on the Turing test:

[Turing] opens with the words: “I propose to consider the question, ‘Can machines think?’” Because “thinking” is difficult to define, Turing chooses to “replace the question by another, which is closely related to it and is expressed in relatively unambiguous words”. Turing describes the new form of the problem in terms of a three-person party game called the “imitation game”, in which an interrogator asks questions of a man and a woman in another room in order to determine the correct sex of the two players. Turing’s new question is: "Are there imaginable digital computers which would do well in the imitation game?

One should bear in mind that scientific methodology was not very formalized at the time. Today, it is self-evident to any educated person that the “judges” would have to be blinded, which is the whole point of the text chat setup.

What has been called “Turing test” over the years is simultaneously easier and harder. Easier, because these tests usually involved only a chat without any predetermined task that requires thinking. It was possible to pass without having to think. But also harder, because thinking alone is not sufficient. One has to convince an interviewer that one is part of the in-group. It is the ultimate social game; indeed, often a party game (haha, I made a pun). Turing himself, of course, eventually lost such a game.

All I can say is that with the level of intelligence current leading AI have, they make silly mistakes that seems obvious if it was really conscious.

For instance as strong as they seem analyzing logic problems, they fail to realize that 1+1=2 <=> 2=1+1.

This connects consciousness to reasoning ability in some unclear way. The example seems unfortunate, since humans need training to understand it. Most people in developed countries would agree that the equivalence is formally correct, but very few would be able to prove it. Most wouldn’t even know how to spell Peano Axiom; nor would they even try (Oh, luckier bridge and rail!)

Far fewer than 1 in 20 defendants get a jury trial in the US. If every defendant insisted on their right to one, then the system would break down for lack of jurors.

Few juries would decide to nullify, since, by and large, Americans believe in punishment.

So the change would be insubstantial.

Self aware consciousness on a human level.

How do you operationally define consciousness?

I find it funny that in the year 2000 while attending philosophy at University of Copenhagen I predicted strong AI around 2035.

That seems to be aging well. But what is the definition of “strong AI”?

I work retail and we’ve had an onslaught of freight

How’s going now? Any noticeable change yet?

I’d be more supportive of fungi independence if they aimed for a democratic republic. Just saying.

Oh Lemmiwinks, Lemmiwinks, …

Ah, you may leave here for four days 11 minutes in space

But when you return, it’s the same old place

The poundin’ of the drums, the pride and disgrace

You can bury your dead, but don’t leave a trace

Hate your next door neighbor but don’t forget to say grace

20 ccs of lasagna, stat!

The high frequencies are the first to go.

Just the German branch of The Coca-Cola Company.

Nah. That’s just propaganda. The copyright people make that stuff up to help their argument against fair use and to drown out the likes of the Internet Archive or the EFF.

In truth, these exceptions don’t matter for Meta. They also offer these models as chatbots; as a service. That brings back all those useless bureaucratic hoops. The exceptions would matter for small players, but the copyright industry has pretty much neutralized them anyway.

A very worrying take by that judge, but it befits Trump’s America.

Market harm is supposed to mean the market for a particular work. For example, when everyone torrent a movie, then that movie will plausibly sell fewer copies. That means there’s less economic incentive to produce movies. That directly undercuts the purpose of copyright.

Me, I think we might be better off without expensive movies, if the price is a censorship infrastructure.

This judge seems to understand market harm to mean that incumbents lose market share. Well, that can happen when new technologies arise. Copyright is constitutionally limited to encouraging new developments. No law in any sector provides for a right to a market share. To the contrary, attempting to secure such a right may be a felony under antitrust laws.

History trivia: Fanta was invented in 1941 in Nazi Germany, when Coca-Cola Germany couldn’t get the original syrup because trade was cut off.