Introduction
I have worked with AI and machine learning since 2018, keeping a close eye on the changing landscape. In 2022 when ChatGPT was released I was testing the model and the API with language tasks and then when the vision models were starting to be released I started with those. The tests I ran were with various open and closed models to see their understanding in both one shot tasks and in context learning to try to understand the strengths and weaknesses of the underlying training data.
I had spent time at SMART Photonics doing manual vision tasks due to small unpredictable production runs in a lab to fab setting, where we would have novel issues arise with every production run due to changing materials and processes. This meant that it was very difficult to automate, but I knew the processes and the available data that is inside one company of this type. Meaning if there was an issue with data, it’s essentially solved within closed silos.
LLaVa 13b, small vision model one shot
The first tests I did were on photonics chips, to see how well language models would be able to recognise them compared to silicon electronic chips which are far more ubiquitous. The results were as expected given that information. When giving the models an image they could recognise the electronics and based output on electronic circuit boards. The smaller models gave less detail and the larger models were able to coherently describe components and their function in more detail.
GPT4 Vision zero shot
After testing chip features and the ability to recognise defects I moved on to using very basic prompts in addition to multiple images for in context learning, essentially using the relation between images as a visual description of what the LLM was looking at in order to affect the output.
LLaVa 13b in context learning
Again the small models had very shallow output and while they could describe the basics, they weren’t able to go into much detail despite them having multiple in context examples. The model above is almost twice as large as the one below and the differences can already be seen here and as you move up in model size.
LLaVa HILU (7b) in context learning
As you moved up to larger models with better and larger datasets the level of detail and the ability to present the output in more structured ways becomes evident. The following models were the early versions of both Google Bard (Now Gemini) and GPT4 Vision.
Google Bard 2023 in context learning
Note the level of detail both models are able to go into, as well as the way they structure and present their output. The ability to specifically follow instructions improved, with one weakness being that they were very verbose in their output. A problem that still persists, meaning the user must specifically ask the model for concise output.
GPT4 Vision in context learning
Magnon - Understanding design patterns in nature
The following pages show excerpts of emerging knowledge of biogenic magnetite. Outputs from LLMs using retrieval of new knowledge is still ‘polluted’ by the enormous amount of base knowledge in medical research (Parkinson's disease and dopamine/neuromelanin). There is a bias towards dysfunction and breakdown of materials.The behaviour of these materials is polluted by the overrepresentation in training data about magnetite in bacteria as related to the earth’s magnetic field. These design patterns should be individually understood, like other differences and similarities between human and plants (phytomelatonin vs melatonin and phyto serotonin vs serotonin). At the time LLM’s could not seem to differentiate well or reason about emerging or sparse knowledge in intelligent ways.
Magnetic nanoparticles and bacteria
The image above contains the typical type of data that was most present in LLM’s when testing, there was deep knowledge of bacterial magnetite and its function in relation to aligning with the earth’s magnetic field.
Most of the data in LLM’s regarding biogenic magnetite was still treating the magnetite in the human brain and its function as speculative. Mirroring the text in the images below. New data just wasn’t well represented in a large enough concentration in the training data to be significant, even with detailed prompting and utilising specific text and images for in context learning. The Language models simply didn’t know what they were 'seeing'.
Early research on biogenic magnetite in neuromelanin
This research continues on the signalling that's potentially involved in biogenic magnetite in the brain, not going much into the material composition. The dominant themes in research were reflected in the outputs of models of all sizes, emphasising both bacterial magnetite and signalling rather than materials.
Early research on biogenic magnetite in neuromelanin
It took a lot of manual research to find the gaps in LLM knowledge. It was time consuming, using Google search and various academic sites directly but eventually specific search terms made it easier to find novel things related to any research topic. The image below is a paper about the material composition of anthropogenic biogenic magnetite in the brain. A very detailed report on in vitro breakdown and buildup of magnetosomes by human stem cells, showing that the design and function of this magnetite is specific to the human brain and differs completely from the bacterial kind that’s used for magnetic field alignment.
Human stem cells in the breakdown and build up of biogenic magnetite
A significant portion of the research literature is also medical in nature, so a lot of outputs were related to neurodegenerative disease and the buildup of different forms of magnetite in the brain related to pollution and the ability of melanin to bind to metals. That function is thought to be protective, occurring with ageing, but it tells us nothing about, and obscures the function of healthy endogenous magnetosomes in the brain.
Memory and magnetic fields
Following the composition and make up of the healthy material in addition to understanding the brain as the materials break down, as with neurodegenerative disease gives a better idea about function. For example the build up of specific materials is associated with both coordination and with higher brain function in children. As those materials are lost to disease the opposite can be seen with a breakdown in higher brain function and coordination with the loss of magnetic material in an organ that relies on electromagnetic coordination and integration to function. Even after the breakdown in materials harmonic devices can enhance function with various electronic, magnetic and musical devices and instruments being examples, such as when dementia patients are played their favourite music.
intergenerational iron 'memory' in bacteria
Once the specifics of behaviour are understood in one organism then it gets easier, or becomes a little more obvious that you can draw conclusions about similar behaviours. The paper above investigates the idea of iron, magnetism and ‘memory’ in simple bacterial cells over generations
LLMs have come a long way since 2023 — deep research and other agentic methods can now handle much more complex work. Whether they can independently surface genuinely novel scientific discoveries from new or sparse data still seems unlikely to me, since a model is ultimately shaped by its training data. My guess is that language models will keep being excellent tools, but the biggest breakthroughs will still come from people using them well, rather than from the models working it out alone — at least until there's a lot more progress on how they handle genuinely novel, sparse evidence.