Giovanni Bricconi

I would like to learn more about machine learning, and I recently came across an interesting ACM journal: Transactions on Intelligent Systems and Technology.

The latest issue (https://dl.acm.org/toc/tist/2022/13/5) contains many papers on federated learning. I have decided to take some notes, not to forget about them. Luckily you can download many of these papers for free if you are interested.

First of all, what is federated learning? Usually, you have a set of input data, classified, and you apply an algorithm to train a model. For instance, you can train a neural network with this data. 

This setting implies that one single entity has access to all the training data, but sometimes this is not possible. For instance, the law can forbid sharing patient data across different hospitals. Also, some counties have laws prohibiting to export personal information outside theirs territory, or protecting consumers’ privacy.

So it is not possible to concentrate all the data in a single place to train a global high-quality model. Models trained just with local data can be less effective, and it is worth finding a way to overcome this limitation.

The approach described in these articles still requires a central coordinator, but it also requires each entity holding the training data to perform model training. From time to time the model parameters are sent from the local entities to the central coordinator. This one updates a global model and sends back the new averaged parameters to each participant. The process is repeated again and again until the training is completed.

Notice that in this way the entities share just parameters and not personal information, overcoming law restrictions. 

Some authors report that federated learning has been originally proposed by Google for the use case of Gboard query suggestion (https://arxiv.org/abs/1610.02527), but I did not read this article.

The first paper I read is: Dimitris Stripelis, Paul M. Thompson, and José Luis Ambite. 2022. Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settings. ACM Trans. Intell. Syst. Technol. 13, 5, Article 78 (October 2022), 29 pages. https://doi.org/10.1145/3524885. It describes synchronous and asynchronous approaches to Federated Learning, and proposes a new algorithm to share model parameters across the participants.

From what I have got, with synchronous learning, each participant trains its model for a fixed number of ephocs and sends an update to the coordinator. When the coordinator has all the updates, it changes the global model that is sent back to the participants. The process then repeats.

If the participants are different, some fast and some slow, the training proceeds slowly and the more powerful sites are underused.

With asynchronous learning, each participant works at their own pace and there is more communication with the central coordinator, each time an epoch is completed. More communication means more energy consumed, and the paper gives some insight into this problem. Another problem is that slower participants continue training on an old model, while the coordinator has a better version, computed using faster participants’ updates.

With their proposal, the training proceeds until a fixed time limit and the updated models are computed and shared across all participants. The paper explains also how the parameters are averaged.

Another interesting point I have learned is the IID/non-IID assumption. IID stands for Independent and Identically Distributed. If all the participants have IID data, it makes much sense to train a model with all of them. If this is not the case, and one participant is different, this can affect the model convergence.

Another interesting paper is: Trung Kien Dang, Xiang Lan, Jianshu Weng, and Mengling Feng. 2022. Federated Learning for Electronic Health Records. ACM Trans. Intell. Syst. Technol. 13, 5, Article 72 (October 2022), 17 pages. https://doi.org/10.1145/3514500.

It provides a survey of existing applications of Federated Learning on Electronic Health Records, to predict in-hospital mortality and acute kidney injury in intensive care units.

Different algorithms are compared, and one can get references and understand a few of their performances. It also describes CIIL cyclic institutional increment learning: a setting where each party trains a model with its data and passes its parameters to the next hospital. The Federated learning approach, with a central entity assembling a common model, outperforms CIIL approach.   

A few paragraphs above I was writing that it is fine to share model parameters, as they do not contain personal information. But is this really true? Can the central server guess personal information from parameter updates? Xue Jiang, Xuebing Zhou, and Jens Grossklags. 2022. SignDS-FL: Local Differentially Private Federated Learning with Sign-based Dimension Selection. ACM Trans. Intell. Syst. Technol. 13, 5, Article 74 (October 2022), 22 pages. https://doi.org/10.1145/3517820 gives some insight on this point.

The idea is to trade model accuracy with privacy: the model update will not be accurate as it could be and will convey only part of the information. The global model will converge more slowly. 

You can use a random variable to decide which model dimensions you want to share with the central coordinator, and at each update just send these instead of all the updates. In the paper, just k of the smallest or largest model parameters are shared.

The last paper I have read so far is: Bixiao Zeng, Xiaodong Yang, Yiqiang Chen, Hanchao Yu, and Yingwei Zhang. 2022. CLC: A Consensus-based Label Correction Approach in Federated Learning. ACM Trans. Intell. Syst. Technol. 13, 5, Article 75 (October 2022), 23 pages. https://doi.org/10.1145/3519311

Not all participants will have the same ability in assigning correctly the labels to training data. Additionally, with Federated Learning, these samples will not be available to other parties, due to privacy restrictions, so it is not feasible to review them centrally before training the model.

Some of these samples may just be errors, but just removing them may also prevent the model to learn new patterns. Anyway using them early during the training can slow down the learning process or lead to model overfitting.

The method proposed in the article keeps some samples outside the training set during the initial learning epochs. Some thresholds are computed and used later to decide whether to admit the samples in later stages or even correct the label assigned to them.

Vivere all’estero comporta non poche complicazioni, specialmente quando bisogna avere a che fare con la pubblica amministrazione. Quando ho dovuto rinnovare la carta d’identità al consolato, e mi hanno spiegato dello SPID, e la cosa mi ha fatto molto piacere.

Molto tempo fa, una decina di anni, avevo cercato di ottenere un codice di accesso al sito dell’INPS e mi ricordo che era stato molto complicato, e che avevo dovuto aspettare una lettera inviata per posta. Avevo ricevuto un estratto conto dei versamenti pensionistici e mi mancava un anno di contributi. Chiaramente non è una cosa urgente da sistemare, visto tutto il tempo che mi manca prima della pensione, ma prima o poi avrei dovuto perderci del tempo. Avevo anche mandato una raccomandata con della documentazione, ma non avevo mai ricevuto una risposta.

Ovviamente dopo tutto questo tempo il vecchio codice é andato perso, e lo SPID mi ha dato una speranza di riuscire a sistemare la cosa, senza dovermi presentare fisicamente in uno sportello in Italia.

Finalmente in ferie, ho trovato un’oretta di tempo per occuparmi della questione, e tentare almeno di fare il login sul sito dell’INPS. Cercando su google ho trovato un articolo di Aranzulla: come ottenere le credenziali SPID. L’ho letto sommariamente una sera, e l’unica cosa che mi ricordavo é che poste italiane ha una app che permette di ottenere lo SPID facilmente.

Bene quindi tentativo numero uno, cercare su appstore l’applicazione PosteID e provare a usarla. Ho cercato nei documenti della carta d’identità elettronica ed ho recuperato la prima e la seconda metà del pin della carta. La prima metà mi era stata data al consolato, la seconda è arrivata per posta con la carta stessa.

Va bene la sicurezza, ma se mi davate il codice in una volta sola, alla domanda, visto che tanto senza carta non ci fai nulla, forse sarebbe stato un po’ più semplice. Comunque, con carta e codici in mano provo a usare PosteID.

Una breve spiegazione descrive cosa fare, basta fare un paio di scelte e ci si ritrova ad inserire il pin della carta e provare a leggerne il contenuto appoggiandola al telefono. Tutto semplice no? Invece errore di lettura della carta, senza altre spiegazioni. Sarà che la mia carta è stata rilasciata all’estero e avrà qualche cosa particolare, in qualche campo? Non è dato saperlo… ho fatto qualche altro tentativo sperando di non avere bloccato la carta, almeno con il PIN ti danno anche il PUK.

Ovviamente avendo dimenticato tutto quello che Salvatore Aranzulla diceva nel suo articolo, ho iniziato a cercare in google ed ho trovato una pagina con una lunga lista di fornitori di identità SPID, se quello delle poste non va bene per me magari qualche altro mi avrebbe potuto aiutare: la lista si trova su come scegliere tra i gestori di identità digitale.

Ho iniziato a scaricare applicazioni sul telefono, ArubaOTP, poi SielteSPID e SielteID.

Prima ho provato con Aruba, ma non ho capito come fare con la carta di identità elettronica, e non ho un lettore di carte per il mio pc… quindi ho lasciato stare.

Nella lista di fornitori c’era banca intesa, ho provato a vedere cosa proponevano, ma non voglio pagare e voglio usare la carta di identità elettronica, dopo una o due pagine ho lasciato perdere.

Dalla lista o provato a vedere qualche fornitore meno noto, ho scelto a caso Sielte, e provato a registrarmi. La procedura è un po’ complicata, e ti si chiede di marcare come letto alcuni documenti di cui uno di 96 pagine vabbè. Tutto andava abbastanza bene sino a quando mi chiedono di inserire una foto non solo della carta di identità ma anche della tessera sanitaria. Per un residente in Italia non è un problema, ma io non ho più una tessera sanitaria perché risiedo all’estero. Ho lasciato perdere.

Stavo per provare Namiral, è su una loro pagina ho finalmente letto che la zecca dello stato ha una applicazione che si chiama CieID, apposta per la carta di identità elettronica. Proviamo.

Wow, la spiegazione era super chiara, ci sono addirittura immagini che spiegano come mettere la carta di identità e dove appoggiare il telefono, e consigli su come tenere fermo il telefono. Inserisco il pin e… questa volta viene letta correttamente! Tutto questo mi ha preso almeno un ora. Ho spento tutto ed oggi ho provato ad accedere al sito dell’INPS.

Incredibile ma vero, sono riuscito a entrare sul sito, perché accanto all’icona dello SPID c’era anche quella della carta di identità elettronica, e quindi ho potuto usare l’app della Zecca. Ancora più incredibile, ora nel mio estratto conto ci sono tutti i versamenti, anche quelli che mancavano; oggi è proprio una giornata fortunata!

Bene, i comuni non possono fare questo servizio di identità, perché fanno già molti servizi. Si è quindi creato tutto un ecosistema privato di fornitori di accesso, ma non è per nulla semplice scegliere quello giusto. Per fortuna c’è rimasta la Zecca di Stato, un grazie incredibile a voi!

Ora mi tocca solo aspettare il 2047 per andare in pensione

Let me introduce my new friend, the electricity counter:

The green block on the upper-left, here in France, is called Linky. Its display is just 2 text lines, quite synthetics, but I have found them useful. There is a lot of bla bla about the contract, but pushing and pushing the buttons it has finally shown something interesting.

It is capable of displaying the instantaneous power consumption and the maximum power provided within a period. It’s not clear how long the time period is, but it has been a piece of valuable information. Are you wondering why?

I just searched on the internet for electricity prices and I have found a table like this on the site https://electricite.net/edf/tarifs

Puissance souscrite	Abonnement annuel TTC
3 kVA	103.49€
6 kVA	136.12€
9 kVA	169.89€
12 kVA	204.03€

The second column gives you the price that you have to pay yearly to have a certain amount of power available to you. If you exceed that level, the green box will cut the electricity… an unpleasant experience.

When I arrived here I have been provided a 12kVA subscription, bit I was quite convinced it was an excessive one; the green box has given me the confirmation. My maximum power consumption is far below 12kVA, and I have managed to save some money stepping to 9kVA. I need some more checks but I am pretty sure I don’t need more than 6kVA, so I can save about 70 euros per year. I am very happy to have spent some time looking at my green friend. For reference, in Italy nearly everybody has 3.5kVA, but they cook using gas…

The third column gives the price by kWh used, and here you get super easily lost in marketing traps. Every company provides its way to compute the rate, and some are very complicated. I have been suggested to chose a rate wich makes you pay different prices during the night and the weekend. I had to take some bills and do some calculations with open office, but I realized that:

1 I am not disciplined enough to use a lot of electricity during the night

2 I have to pay a special subscription to have time-based rates, and I spend more than what I save

It may seem complicated, but if you resist to the temptation of finding the more economical rate on the market, and you just analyze 2/3 mainstream offers, you will be able to check your subscription too in just a couple of hours.

I said above that my green friend also show the instantaneous power consumption. When I was playing with it was a weekend, and at home there was nothing special switched-on. No air conditioning, nothing cooking, no oven, no dishwasher…. an the box was telling me: you are using 2300W. What??? With nothing turned on!!

You can see in the picture that the electric cabinet is filled with switches, each of them excludes something: the problem is that it is not at all clear what it does exclude, and there are no indications on the switches.

I started playing with the switches until I have found one that tuned off about 1800W consumption. I did not have idea at all of what I had turned off, it has taken a day to discover it. This in the end what I have found after some experiments:

The water boiler has not been wired correctly: it should be on a specific circuit, working only during the night. In my house it was running randomly all the day, making my electricity consumption explode.

The cooking plate is consuming 200W just for being connected to the electricity, even if it is turned off! Just turning off its switch made me save the equivalent of 2 old very bright bulbs working all the day.

The air conditioning, like the cooking plate, consumes also if it is turned off… so during spring and autumn there is no need to keep it connected, and its switch can be turned off.

I was quite bored of experimenting and there is not so much I can do to save more electricity, so that is enough for me. I won’t do more experiments but the time spent was definitively worth it.