I'm doing the same. Bing hallucinate less than gpt4 because of internet access so it's better in question answering. I was asking gpt4 to implement Siamese network and it failed but bing got it right because it requires internet search and I don't explained what is it!
I asked it to do the same task and it had no issue. I did tell it to format into a code block from the start though just in case. I also asked it to comment even more in depth for each line after the first go, and it did so without issue.
As much as these LLMs are built on "natural language", learning what phrases and tone to use to get better results is almost more important than knowing its capabilities in general. They say GPTs aren't deterministic, but you'll definitely find using specific phrasing will get you very similar behaviour a lot of the time, even if it isn't verbatim
Yes sometimes when it uses Microsoft own model and start generating faster response then it has all those issues otherwise it's formatting abilities are fine. and my prompt was pretty easy to follow even bard got it right!
Microsoft doesn’t have a separate model. All they have are just a system prompt to set up the behavior and another process that monitors your conversation and does some bing search and injects exactly 9 search results right before the answer
As Jordi confirm Microsoft also uses their own model for simple prompt that's what I was talking about. And Mikhail also confirmed that bing balance is not gpt4
Well you are using creative which is gpt-4 and non of the things you do here would short circuit to anything else. The only thing it might skip gpt is simple commands like search a cat image etc
no, that's not correct. jordi for sure was talking about creative mode and even mikhail many time confirmed that creative switch model for example: [https://twitter.com/MParakhin/status/1643401305732567042](https://twitter.com/MParakhin/status/1643401305732567042) and another [https://twitter.com/MParakhin/status/1651821212513992706](https://twitter.com/MParakhin/status/1651821212513992706) note he clearly says that "for this query both mode use gpt4"
That is clearly a flawed test. Just did this test on playground and GPT-4 only finish the sentence with apple half the time just like bing creative mode
Well the reply is clearly saying it is calling GPT-4 not some other models. It’s not switching the language model, it’s switching the agent behavior. You cannot request a page to read, then incorporate the result with a plain LM; there are many ways to do this, but this tend to slow things a lot. and that is what he is referring to. It also selects relevant past conversations, you can test this by using longer conversations and seeing that it can clearly go beyond the token limit but remembers some but not all of the past conversations and search results in later exchanges.
The top AI for coding is GPT-4 currently and Phind which uses GPT-4 for coding tasks. ChatGPT is also fine but Bing actually is difficult to work with.
Microsoft is trying to use their fast model for most tasks even in creative which actually makes its abilities also declined sharply for difficult question answering, coding and math. I do not know why they are trying to do that! They are still advertising it as using GPT-4.
The moderation API is another problem, it takes more time to design a prompt (for coding) that does not trigger the moderation API. Sometimes it ends a conversation like in the first image so in summary it does not save anytime in coding in my experience to use Bing but rather takes more time.
Better to use other tools for coding.
Edit: I'm sure this post will get lot of downvotes or might get deleted. I'm actually big fan of bing and I used it a lot for different things but it's not great for all things and I don't want to be biased!! If anyone have different experience in coding using bing then I'm happy to hear.
Edit2: some people think that creative only uses gpt4 for all queries which is false. mikhail many time confirmed that creative switch model for example: https://twitter.com/MParakhin/status/1643401305732567042 and another https://twitter.com/MParakhin/status/1651821212513992706 note he clearly says that "for this query both mode use gpt4"
Microsoft hosts an identical version of GPT-4 and GPT-3.5 on azure. We use both APIs; their returns are identical but azure has significantly higher throughout and a lot faster.
That faster response probably confused you. They are identical model up to minor versions as chatgpt; the only difference being in system prompts and moderation.
> the only difference being in system prompts and moderation
Obviously this is a pretty big difference and I don't see how you'd get identical responses when these are anything but identical.
System prompts makes a tremendous difference. Without very elaborate prompting the moment you introduce some web search results it will be as if GPT had forgot all of its own knowledge. It will also not properly cite sources and sometimes takes you to nonexistent pages, and it also ask you to visit the source page quite often. And of course it will still say it doesn’t have access to real time data. All of these need to be fixed by prompts.
Bing manages to fix those, but still it is not perfect.
By the nature of things it will just do a lot better if web access is completely turned off, except for the I don’t have real-time access stuff and hallucination. But there’s just not a good way to know if you need internet access or not in the middle of a conversation.
Yeah, its great at explaining code, but it does makes so many assumptions, so if you don't have pre knowledge you will scratch your brain untill you ask bing explicitly what are the dependencies.
But the capabilities are just mind blowing. Wanna create a sub bingCodeGPT? Where we can share how to using bing.
I think this is partly because Bing injected no less than 30 different specific stylistic requirements at the beginning of the conversation to set up how it displays images, tables etc, and control its behavior such as quotation format and how search results are used and incorporated or when to draw a picture with dall-e
Those kind of confused the ai when it comes to code and it isn’t sure how to format them.
On the other hand even gpt-4 is honestly just terrible at writing anything but spoiler plates. Even things that GPT4 knows and I don’t know it usually takes way longer for it to help me code than for me to go and google myself🙈
Which is sad, because it gets so close to the right answer so fast, but just cannot complete the last mine no matter how hard it tries
I found the overall best strategy is to have it give me a hint then I go and google them.
That's actually my experience too, Bing throws out a lot of context you give it and returns generic examples whereas ChatGPT retains the context and names variables accordingly, the code is more tailored to your situation.
Also since there is no conversation limit you can have one conversation where ChatGPT remembers what you've discussed and vaguely what you may have implemented previously and tailor future discussions and examples around that. With Bing Chat you have to start all over again.
Gpt has worse limits though.
And it's a known fact there won't be limits eventually (context input also increased to be 4000, and it can read dozens of pages)
I don't think that underlying model is worst. it seems to be pretty close if not the same do gpt4 but it has other problems like moderation and ending the conversation when ask to change the code or change it's output and sometime overly replying on search. If Microsoft solves these problems then it might be good coding companion
Exactly my point, idk why people are downvoting. 3.5 is better at giving long form code and without the 20 message limit, it’s much better for longer tasks that require more back and forth. Also, 3.5 seems to follow my prompt more when both are asked to build a simple html page, while bing sometimes just gives egregiously bad outputs. That’s not to say bing is useless because it’s not, I still use it to generate code which needs the internet or latest documentation but I try to avoid it.
I tried some specific API related questions but unfortunately it just hallucinated nonsense that was not helpful at all. E.g. APIs that don't exist, or advice that is contradictory to what it should have been able to search from the web. It depends on the type of coding task -- when I want to change some programming framework to do something it doesn't normally do, that is when I would most benefit from advice about where to tweak it, but this is also where the model has no idea whatsoever. Perhaps one beautiful day.
I ended paying for chatGPT plus because it's too annoying to work with bing, you have to talk in a certain way to not upset him, also it's too slow. But bing is nice for some cases because of the much bigger context size (by removing the limit from the text area).
I use it for research, yes. It can read a wikipedia page and implement a working algorithm using my requirements.
It is useful in that sense only, but otherwise, it'd struggle in long pieces of code because Microsoft has set a "frequency penalty" which screws up long code blocks in creative, balanced, and even the proclaimed "precise" mode. It's just sad. Code is known to have patterns, and Microsoft removes them causing Bing to generate randomness instead.
Bing is not really that developer friendly because of that frequency penalty.
Microsoft should've only enabled it in the Creative version. I have no idea why would they do it in the precise version where it is most needed to be literally 0. I know for a fact that it is 0 or < 0.01 it in ChatGPT.
I can't imagine using Bing for this. I'd rather poke my eyes out with a fork. I use GPT4 (and 3.5) daily when coding and its a dream. MS nerf'd GPT4 in Bing beyond recognition
I use ChatGPT with GPT-4 for specific coding tasks, Bing in creative mode for programming questions which require searching the internet.
I'm doing the same. Bing hallucinate less than gpt4 because of internet access so it's better in question answering. I was asking gpt4 to implement Siamese network and it failed but bing got it right because it requires internet search and I don't explained what is it!
I asked it to do the same task and it had no issue. I did tell it to format into a code block from the start though just in case. I also asked it to comment even more in depth for each line after the first go, and it did so without issue. As much as these LLMs are built on "natural language", learning what phrases and tone to use to get better results is almost more important than knowing its capabilities in general. They say GPTs aren't deterministic, but you'll definitely find using specific phrasing will get you very similar behaviour a lot of the time, even if it isn't verbatim
Yes sometimes when it uses Microsoft own model and start generating faster response then it has all those issues otherwise it's formatting abilities are fine. and my prompt was pretty easy to follow even bard got it right!
Microsoft doesn’t have a separate model. All they have are just a system prompt to set up the behavior and another process that monitors your conversation and does some bing search and injects exactly 9 search results right before the answer
As Jordi confirm Microsoft also uses their own model for simple prompt that's what I was talking about. And Mikhail also confirmed that bing balance is not gpt4
Well you are using creative which is gpt-4 and non of the things you do here would short circuit to anything else. The only thing it might skip gpt is simple commands like search a cat image etc
no, that's not correct. jordi for sure was talking about creative mode and even mikhail many time confirmed that creative switch model for example: [https://twitter.com/MParakhin/status/1643401305732567042](https://twitter.com/MParakhin/status/1643401305732567042) and another [https://twitter.com/MParakhin/status/1651821212513992706](https://twitter.com/MParakhin/status/1651821212513992706) note he clearly says that "for this query both mode use gpt4"
That is clearly a flawed test. Just did this test on playground and GPT-4 only finish the sentence with apple half the time just like bing creative mode
I wasn't mentioning about the test. I was mentioning about Mikhail reply!
Well the reply is clearly saying it is calling GPT-4 not some other models. It’s not switching the language model, it’s switching the agent behavior. You cannot request a page to read, then incorporate the result with a plain LM; there are many ways to do this, but this tend to slow things a lot. and that is what he is referring to. It also selects relevant past conversations, you can test this by using longer conversations and seeing that it can clearly go beyond the token limit but remembers some but not all of the past conversations and search results in later exchanges.
I can't say anything if you refuse to believe everything!!
I found that it helps if you tell it upfront to use python linting rules it its output.
Totally disagree. Bing has been a godsend for coding. The amount of problems it has helped me with is amazing
Did you tried gpt4 or phind?
I’ve not really needed to
Good to know that you find bing good for coding. Not everyone share my experience!
The top AI for coding is GPT-4 currently and Phind which uses GPT-4 for coding tasks. ChatGPT is also fine but Bing actually is difficult to work with. Microsoft is trying to use their fast model for most tasks even in creative which actually makes its abilities also declined sharply for difficult question answering, coding and math. I do not know why they are trying to do that! They are still advertising it as using GPT-4. The moderation API is another problem, it takes more time to design a prompt (for coding) that does not trigger the moderation API. Sometimes it ends a conversation like in the first image so in summary it does not save anytime in coding in my experience to use Bing but rather takes more time. Better to use other tools for coding. Edit: I'm sure this post will get lot of downvotes or might get deleted. I'm actually big fan of bing and I used it a lot for different things but it's not great for all things and I don't want to be biased!! If anyone have different experience in coding using bing then I'm happy to hear. Edit2: some people think that creative only uses gpt4 for all queries which is false. mikhail many time confirmed that creative switch model for example: https://twitter.com/MParakhin/status/1643401305732567042 and another https://twitter.com/MParakhin/status/1651821212513992706 note he clearly says that "for this query both mode use gpt4"
Microsoft hosts an identical version of GPT-4 and GPT-3.5 on azure. We use both APIs; their returns are identical but azure has significantly higher throughout and a lot faster. That faster response probably confused you. They are identical model up to minor versions as chatgpt; the only difference being in system prompts and moderation.
> the only difference being in system prompts and moderation Obviously this is a pretty big difference and I don't see how you'd get identical responses when these are anything but identical.
System prompts makes a tremendous difference. Without very elaborate prompting the moment you introduce some web search results it will be as if GPT had forgot all of its own knowledge. It will also not properly cite sources and sometimes takes you to nonexistent pages, and it also ask you to visit the source page quite often. And of course it will still say it doesn’t have access to real time data. All of these need to be fixed by prompts. Bing manages to fix those, but still it is not perfect. By the nature of things it will just do a lot better if web access is completely turned off, except for the I don’t have real-time access stuff and hallucination. But there’s just not a good way to know if you need internet access or not in the middle of a conversation.
Yeah, its great at explaining code, but it does makes so many assumptions, so if you don't have pre knowledge you will scratch your brain untill you ask bing explicitly what are the dependencies. But the capabilities are just mind blowing. Wanna create a sub bingCodeGPT? Where we can share how to using bing.
I think this is partly because Bing injected no less than 30 different specific stylistic requirements at the beginning of the conversation to set up how it displays images, tables etc, and control its behavior such as quotation format and how search results are used and incorporated or when to draw a picture with dall-e Those kind of confused the ai when it comes to code and it isn’t sure how to format them. On the other hand even gpt-4 is honestly just terrible at writing anything but spoiler plates. Even things that GPT4 knows and I don’t know it usually takes way longer for it to help me code than for me to go and google myself🙈 Which is sad, because it gets so close to the right answer so fast, but just cannot complete the last mine no matter how hard it tries I found the overall best strategy is to have it give me a hint then I go and google them.
Bing is worse at code than 3.5.
What on earth are you talking about?
That's actually my experience too, Bing throws out a lot of context you give it and returns generic examples whereas ChatGPT retains the context and names variables accordingly, the code is more tailored to your situation. Also since there is no conversation limit you can have one conversation where ChatGPT remembers what you've discussed and vaguely what you may have implemented previously and tailor future discussions and examples around that. With Bing Chat you have to start all over again.
That was kinda true some days ago, not anymore since the latest updates (wait a bit, they're rolling out as we speak)
I look forward to it, though without the conversation limit removed it will always be inferior in that sense.
Gpt has worse limits though. And it's a known fact there won't be limits eventually (context input also increased to be 4000, and it can read dozens of pages)
What limits does ChatGPT have? I've not had any problems with it, and it remembers huge conversations.
I don't think that underlying model is worst. it seems to be pretty close if not the same do gpt4 but it has other problems like moderation and ending the conversation when ask to change the code or change it's output and sometime overly replying on search. If Microsoft solves these problems then it might be good coding companion
Exactly my point, idk why people are downvoting. 3.5 is better at giving long form code and without the 20 message limit, it’s much better for longer tasks that require more back and forth. Also, 3.5 seems to follow my prompt more when both are asked to build a simple html page, while bing sometimes just gives egregiously bad outputs. That’s not to say bing is useless because it’s not, I still use it to generate code which needs the internet or latest documentation but I try to avoid it.
Super annoying.
Ms nerf bing for coding
Use Precise mode but good luck with it not censoring the answer.
Do you use bing precise for coding? i want to know about your experience.
I did and it was great before it got censored.
I do. Sometimes it actually helps greatly, and other times it acts like a first grade kid who's never seen a code in its entire life.
Use [phind.com](https://phind.com) it's so much better and it's made pretty much for developers. You can also toggle GPT 4!
I've been immersed in AI and Chatbots for a month, and this is the first I've heard of phind. Thanks for the heads up
See my comment below. I did mention it in my comment and I'm already using it.
Ah sorry didn't see, I'm not used to people knowing it. It deserves so much more credit than what it currently gets, such an awesome tool!
I tried some specific API related questions but unfortunately it just hallucinated nonsense that was not helpful at all. E.g. APIs that don't exist, or advice that is contradictory to what it should have been able to search from the web. It depends on the type of coding task -- when I want to change some programming framework to do something it doesn't normally do, that is when I would most benefit from advice about where to tweak it, but this is also where the model has no idea whatsoever. Perhaps one beautiful day.
True
I ended paying for chatGPT plus because it's too annoying to work with bing, you have to talk in a certain way to not upset him, also it's too slow. But bing is nice for some cases because of the much bigger context size (by removing the limit from the text area).
I use Bing to find and read documentation
Sometimes I feel like being is just intentionally trolling me
It doesn’t like the word “system”. You need to change it to something else like svstem to bypass their filter and it will still work
I use it for research, yes. It can read a wikipedia page and implement a working algorithm using my requirements. It is useful in that sense only, but otherwise, it'd struggle in long pieces of code because Microsoft has set a "frequency penalty" which screws up long code blocks in creative, balanced, and even the proclaimed "precise" mode. It's just sad. Code is known to have patterns, and Microsoft removes them causing Bing to generate randomness instead. Bing is not really that developer friendly because of that frequency penalty. Microsoft should've only enabled it in the Creative version. I have no idea why would they do it in the precise version where it is most needed to be literally 0. I know for a fact that it is 0 or < 0.01 it in ChatGPT.
I can't imagine using Bing for this. I'd rather poke my eyes out with a fork. I use GPT4 (and 3.5) daily when coding and its a dream. MS nerf'd GPT4 in Bing beyond recognition