The ugly solar panels odyssey 2
A lot of time has passed since my previous post: now my solar panels are installed and fully operational, but it has been really difficult to reach this stage. Why? Because people many people just say caring about environment, but in reality they care much more of aestetic.
In France you just have to write a request if you need to install a TV antenna or an electric plug to recharge your car, but if you want to install solar panels on your roof it is a completely different procedure. You first need to get an agreement from your coproperty, and installing solar panels is not a priority. You need to wait for the yearly meeting to discuss your request. You need also to be ready to discuss your request! Solar panels are just rectangular black panels that you need to align on your roof, but roughly one half of the owner were just unhappy because my roof would have been different from the others. Some were saing: “if we let you install the panels, then somebody else will maybe want to install them and will chose a different model, and we will end having all the roofs different”. They just don’t care if theirs electricity is made with uranium, petrol or gas, just they must see something pleasant when they wake up in the morning. But why a solar panel is worst than a satellite antenna? An old lady was really upset and was saying me to take care of the my child’s energy and not of silly green energy.
The condo board president tried in many ways to let me retire my request, saing that would have been better to put the panels only on the side of the roof that is less visible from the others. Unfortunately this is technically bad in my case, because it would have reduced a lot the hours when the energy is produced. Moreover I would have had to wait for one year to present again my request. Luckily for me, my request passed for just one vote! Notice that my installation is very small, just 16 squared meters, and just 8 squared meters visible to the others.
Done with the condo, I had to make a request to the town. Any Mayor has is own preferences, in some villages they just don’t want installations that change the “skyline”, so you can forget about solar panels. Luckily for me in my village is not an issue and I had to wait just a couple of months to get the reply. So it passed more that one year since the moment I decide to start my project to the time I got my project approved. Then I had to wait for the supplier to be ready, and since there have been many requests I had to wait for months.
All the burocratic procedure is extremely slow also. Once the panels are in place you need to wait for a paper from the grid authority stating that the instalation is ok, and only with that you can ask to a distributor to sell your surplus electricity. It has taken 6 months more to finish the procedure and I got my first money only this March. Getting subventions also requires providing a lot of papers, and the instructions were not clear. In the end there were 2 kind of subventions but nobody told me that they were mutally exclusive! I still don’t have a reply for one of the procedures 2 years after my request was filed.
I think 3 years passes since I started my project. One third of my electricity bill disappeared, and I am happy that everything is working fine. What is very disturbing for me is all this fake ecologism: everybody sais that green is good, but then nothing in the system helps the very few of us that wants to make a change to realize theirs projects. I did not disturb anybody with my project, it’s all my money except 10% subvention I got, and it will take me 10 years to reach the break even point, but was my decision. It is unfair that so many obstacles are placed on the road.
AI sophia-summit.com
Hello, it has been a while since I have written my last post: sometimes the time for things that you really love is very limited. This week I had the privilege of attending the artificial intelligence summit, organized by the cote d’azur university. Many professors, from different domains, have illustrated how deep learning has become pervasive and fuels innovations. Here you see the photos I collected. Go to the web site https://sophia-summit.com to get the references, if you are interested.
ChatGPT4 Omni
Today I wanted to look for another article on artificial intelligence, just to learn something new. During this week I was looking for the prices of the various chatGPT models available in Azure, and I saw that the cost of chatGPT 4o were significantly below the others chatGPT 4 models. I wanted to understand what is this new model, and I have found this video:
Will we ever be able to understand if we are speaking to an human or to a machine since today? Seriously you have to watch the video, frightening and amazing at the same time.
Introduction to Neuroscience
There is a nice book, available for free, if you are curious about how the nervous system works. Valerie Hedges has written it for undergraduate students, she is assistant professor at the Michigan State University. The book is available on the Open Textbook Library, in various formats: I just read it on-line chapter by chapter, but you can download a pdf.
It is very easy to read the book, no background knowledge is required. All the terms are introduced slowly, and there are a lot of pictures to illustrate the concepts. You will learn which kind of cells compose the brain, their different purposes. Neurons are described in theirs various parts, and you will learn about the ions that make the electrical communications possible. The ion pumps are described in a long chapter, and it is really amazing to see how complex is our nature. All our thoughts comes from an infinite set of small cells, receiving signals from many other cells but just producing a single output for the others that connects. Ions suddenly moves across the “skin” of these cells causing electrical communications, in a very ordinate way. All is based on generating impulses, at various frequencies, without consuming much energy.
I was curious to know how the connections between the neurons grows or shrinks (synaptic plasticity), unfortunately I did not find this in the book. Anyway I learned a lot, an the reading was really pleasant. Thank you Valery for having written and published this book for free, you did a great job.
Leaky Integrate-and-Fire
About one year ago I attended a conference where one professor presented the work of her team on simulating real neurons. I was curious about how body neurons can be modeled and I have taken the opportunity of a long train journey to browse a bit.
I stared my search looking for “Spiking neural network”. The most surprising thing I learned is that real neurons seems do nothing most of the time. Suddenly they can fire a spike to alert other neurons, but usually they seem idle. The frequency at which they fire is quite low, and this is probably the reason why the brain consumes so few energy. If you search on google you will quickly see that this frequency is usually less than 1kHz, usually much less. By converse, the “artificial neurons” we use in deep learning models fire continuously and synchronously at gigahertz rates: every parameter need to be evaluated at each clock signal, the output can be zero but we need to recalculate the output of each neuron in the network.
Biological neurons seems modeled as a Capacitor with a reset circuit close to it. Each time an input spike is received, the capacitor charge increases. When the charge is high enough, a spike is emitted and the charge is reset. A more realistic model, the leaky integrate-and-fire, puts also a resistance in parallel of the capacitor. It’s role is to discharge slowly the capacitor, to clear the effect of old spikes received too long ago. Input spikes are significant only if they are received in a short time interval.
On this tutorial you will see a good presentation that describes the models and the formulas: https://compneuro.neuromatch.io/tutorials/W2D3_BiologicalNeuronModels/student/W2D3_Tutorial1.html
A biological neuron is composed of dendrides, which are root like structures that receives the inputs from other neurons. The dendrides enters the body of the neuron cell, from which an axon exits: the axon is the output connector that will propagate the spike to thousand of other neurons. Axons and dendrides do not touch directly, they are connected via synapses.
The leaky integrate-and-fire model consists in this formula:
The output current of the neuron depends on 2 components: the leaky part is on the left (with the R internal resistor) and the input driven part with the C constant. u_rest is the tension at rest of the output, when we have not enough input spikes.
The presentation is suggesting some other interesting points.
The spikes are all the same: the shape is the same, in 1-2 ms the spike ends. Seems that it is just important that the spike happened, there is not a concept of high or low spike. By converse artificial neurons have an output that is a real number, maybe normalized but with variability, not just 1 or 0.
There exists more sophisticated models that can explain another phenomenon called adaptation. Suppose you present the same constant input to a neuron, with the leaky integrate-and-fire model the neuron will charge, emit a spike, recharge, emit another spike… So it will generate an output sequence with a specific frequency. Seems that nature do not likes this behavior: a constant signal do not bring much information, real neurons will emit spikes, but with a frequency that decreases. The spikes will be there but less and less frequent.
But how do biological neuron learn? with artificial neurons you have algorithms that uses the gradient and adapt the neuron’s coefficients to learn producing the correct output. I ask myself if a model exists to explain how the dendrides and axon change in response to all these spikes. Maybe a good topic to explore in another train journey.
Large-language-model as a Judge
It is amazing what large language models can do with text, and with libraries like LangChain it is trivial to create chatbots. You can create a small application that uses your internal documentation to answer questions in few hours. I did it with my docs, and works pretty well even if I am not an expert of LLM and machine learning. It is robust, without any tuning you get something working.
But suddenly you start to see something wrong. For instance, I stated in the prompt that I wanted to know the source of every answer. Nothing bad in it, no? To simple questions like “what is the XYZ framework” the creative LLM machine started replying “The XYZ framework does this and that and has been created by Mohammed…”. Ohhh, the word SOURCE means also author, and in each page I have the AUTHOR. I don’t want that the tool starts finger-pointing people. Same happened with an incident post-mortem page: “According to Sebastien, you should check the metric abc-def to verify if the system is slowing down…”
Given the context the answers were correct, but somebody may not be happy of what the machine can tell, and I don’t want to deal with personal identification information at all. Mine is an internal tool, if it was a public site it would have been a real problem.
Now the question are: how can I be sure I have solved this issue? how can I be sure I won’t introduce them again? In the end the output of the chatbot is random and is in natural language so it is not a trivial problem where you can say for question X the answer is exactly Y.
An idea would be to create a panel of questions and prompts to validate the answers: so I won’t evaluate each single answer myself but I will trust the LLM to be able to decide on its own output. This because of course I do not want to spend my time in checking hundreds of questions each time I change a parameter.
Actually evaluating a chatbot is much more complex: I started looking for articles about that and I have found the Ragas framework and this article:
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. 37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks. Lianmin Zheng et al.
The authors explored the idea of using LLM as a judge, comparing the answers of different LLM. In their case they compare different models, but I think this could be useful also to evaluate the answers of different versions of your own chatbot.
In their case they have some questions and ask ChatGPT 4 to decide if the answer of system X is better than that of system Y. In my case I could change some parameter of my chatbot, for instance the number of context document used, and decide which is the best setting.
In theirs article they describe various approaches:
- Pairwise comparison. An LLM judge is presented with a question and two answers, and tasked to determine which one is better or declare a tie.
- Single answer grading. The LLM judge is asked to directly assign a score to an answer
- Reference-guided grading. In certain cases, it may be beneficial to provide a reference solution if applicable.
I do not like much the second and third approaches: asking to get a vote can result in high variability in the answers, and providing reference solutions is hard because you need to prepare these solutions.
Concerning the pairwise comparison the authors highlight some nontrivial issues:
The LLM can prefer the first answer to the second one just because of the position. So you should accept that X is better of Y only if swapping the answers position leads to a consistent result.
LLM could prefer a verbose answer… the more content the better! this seems difficult to address
Self-enhancement bias. for instance ChatGPT 4 favors itself over other LLM in 10% of the cases. This won’t affect much my chatbot as I won’t switch from one model to another. I have just ChatGPT 3.5 or 4 to choose and it is evident which one is the best.
Limited logic math capabilities… LLM are still limited in these domains, the judge will not be good enough in this case.
In the end ChatGPT 4 seems a good enough judge so I plan to use pairwise comparison in future as a method to assess the quality of a new chatbot release.
FAISS, Chroma and Pinecone in action
Developing RAG (retrieval augmented generation) applications requires you to process your documents and store them along with embedding vectors in a sort of database. You need to store the document, because you want to retrieve the exact text that will help the LLM crafting an answer. Nowadays you may not want to do textual search on your documents: leveraging some LLM model you can map a text to a vector of real numbers, that represents the document position in a sort of euclidean brain space. In this space “the bank of the river” and “the bank downtown opens at 09:00” will be mapped to different places, and will be clear that the 2 phrases speaks about different things even though the word “bank” appears in both. One of such embedding model is text-embedding-ada-002.
When the user asks a question, the question is mapped to a real number vector and you have to search the documents more similar to that vector. This requires some sort of specialized database. Looking at LangChain documentation I found that there are at least 3 different databases to explore: FAISS, Chroma and Pinecone.
I searched for a comparison between them: I did not find it but I come across this interesting video Vector Databases with FAISS, Chromadb, and Pinecone: A comprehensive guide. It will not teach you about the differences between the databases, but it will show you how to use theirs API, and tell you about the difficulties you may have installing them. Happy watching!
ReLu or not ReLu?
The ReLu activation function is very popular in neural networks, but many more activation function have been proposed after its introduction. This paper analyzes them from the theoretical point of view:
Deep Network Approximation: Beyond ReLU to Diverse Activation Functions. Shijun Zhang, Jianfeng Lu, Hongkai Zhao
Journal of Machine Learning Research 25 (2024) 1-39
There is a real zoo of activation functions: LeakyReLU, ReLU2, ELU, CELU, SELU, Softplus, GELU, SiLU, Swish, Mish, Sigmoid, Tanh, Arctan, Softsign, dSiLU, SRS… The paper defines some families of activation functions called A1,k ,A2 and A3 that are supersets of all those functions. These sets include also all the functions you can obtain doing translations or reflections. Surprisingly it has been possible to demonstrate that you can approximate as much as you want the output of a ReLU network with any of the listed activation functions, provided that you can afford building a network 3 times wider and 2 times deeper. More formally
Scared? Me too, but at least you know that you can play a bit with the activation and there is a bound to the damage you can make. Moreover if you pick just functions very similar to ReLU, (ELU, CELU, SELU, Softplus, GELU, SiLU, Swish, and Mish) the boundary is much more tight.
RAG Applications
As non English native speaker, it is always funny to try to understand the meaning of a new acronym, this time it is the turn of RAG Applications. I knew about “rag time” music, like this Ragtime Sea Shanty, and searching on the dictionary I saw that RAG stands for a piece of old cloth, especially one torn from a larger piece, used typically for cleaning things. Funny to see the above kind of picture associated to Research Augmented Generation (RAG).
What does Research Augmented Generation actually means? It is a technique to generate content using AI tools like chat GPT: instead of just asking the LLM “write a documentation page on how to shut down a nuclear reactor” you propose a prompt like this “write a documentation page on how to shut down a nuclear reactor rephrasing the content of these 10 pages: page 1. The supermega control panel has 5 buttons…”.
What is the advantage of including in the prompt these examples? The large language model (LLM) may not know anything of your domain for various reasons: maybe you want it to write about something to recent, that has not been used to train it. Another reason may be that you have a very specific domain, where the documentation is not public and therefore the LLM has no knowledge of that.
In my case my team has a private web site where we host our internal documentation: we describe there API, troubleshooting guides, internal procedure, meeting minutes, etc. The LLM does not know anything about our project, so it is impossible that it can provide detailed answers about it.
But how can you retrieve the right 10 pages to embed them in the LLM prompt? If you are familiar with search engines like Solr or ElasticSearch you may have many ideas right now: just export from the site the html text, preprocess it with beautifulsoup to convert it into text and then index it…
Today you can do better than that. Now you have not just LLM that answer questions, you have also models that compute the embeddings of a text like these from OpenAI. You take an html page from your site, you split it in multiple pieces, you ask to get the embeddings of this text and you store them into a specialized engine like ChromaDB. When a question is asked, you compute the question embeddings and ask ChromaDB to give you the 10/20 most similar contents it knows and you format a prompt for ChatGPT that contains those snippets.
It is amazing: we started chatting with our documentation! We can ask how to get a security token, which are the mandatory parameters of a specific API, on which storage accounts we are putting that kind of data…
Why do you need to split your contents in snippets? Simple, an LLM prompt has a size limit, you cannot just write each time a prompt encyclopedia and have an answer… chatgpt 3.5 accept fewer words but is very fast, when trying the 4 we had to wait a lot before seeing the full response. Also you need to consider the price of using the 4 instead of the 3.5.
Will you trust the LLM company for not collecting your private material and using it later as they want? It is much better if you read the user agreement! At least you do not sent the whole material at once to fine tune a new LLM model.
Some frameworks can help you doing this kind of application: Microsoft’s Semantic Kernel and LangChain. I tried the first one, but personally I am not very happy of Python support, maybe better to start with LangChain the next time.
World AI Cannes Festival 2024
The exhibit has been held between February 8th and February 10th in Cannes, French Riviera
While 2023 edition was highly democratized, this time the organization decided to allow access to the interesting speeches only to paying tickets. The company I work for has also bought less ticket so, sadly, I did not profit as much as I wanted of this event. The organization also changed the mobile app they had the last year: in 2023 was so easy to find what was going on across the exhibit, and this time it has become a mess: check each single room to see what was planned… I hope in 2025 they will change their minds.
One interesting speech I saw was “NeuroAI: Advancing Artificial Intelligence through Brain-Inspired Innovations”. The speakers described us some interesting research paths about building more efficient AI hardware. The human brain consumes a ridiculous amount of energy, comparable to a candle light, by converse the modern hardware is really energy-intensive, requiring hundreds of watts to operate. To make the things worst, the current trend is just to build bigger models using more hardware and more training data. It will be impossible to keep this trend, as the energy demanded will quickly become incredibly huge: 85 TWh by 2027 according to this scientific american article
More efficient hardware could probably mimic what the brain is doing: natural neurons fires at some incredibly low rate, just 30Hz. Neurons receive signals, and aggregate them producing an output only once a certain threshold is reached. The output spike does not last for long time and does not consumes that much energy.
It is possible to use some transistors to implement a similar behavior, this could become an unit that can be cloned million of times realizing in hardware an huge array of neurons. Training these devices wold then mean let them learn the correct thresholds and input connections. By converse current architectures require to continously compute values for each neuron output even if the output is not required.
One more energy-intensive task is moving the data around: current architectures have specialized memory units and computation units. The required data has to be moved in and out from the memory to the GPU and back. This requires a lot of energy.
IBM did some experiments, and seems using this neuromorphic patter gives some advantages over traditional LSTM architecture. You can visit Stanislaw Wozniak page to find some references.
I then followed some few other presentations, and some commercial demos. AMD’s one about realtime video processing was both appealing and frightening. You can buy now devices that can process video input streams at up of 90 frames per seconds. You can buy a software stack that allows you to view this camera flow and decide that on some video areas you want to perform some actions: for instance you put a camera watching a car access gate and decide that you use a region of the view to search for a car registration number. You could stack these operation on the remote device or turn you attention on a central system that now just receives events such has the car xx123yy has passed here at 12:05.
The guy setting up this stack of operation does not need to be a rocket scientist, a nice ui can help him setting up his/her video driven application. Moreover you could react to events and change the pipeline of some other cameras to perform other operations. In Singapore parkings facilities they are using this to detect free parkings and guide VIP customers to theirs places… or maybe to avoid non vip customers to pick reserved slots. So far so good, but what does prevent some malicious entity from having cams looking for a specif guy, and being reprogrammed to detect his activity while moving around a building? Do we have to fear that a cam at work place starts detecting how many times we go to the toilets? Unfortunately it does not seems anymore sci-fi… Field Proven Smart City Video Analytics Solution
Giovanni Bricconi 
