Lessons from open source in the Mexican government [LWN.net]

Benefits for LWN subscribers

The primary benefit from subscribing to LWN
is helping to keep us publishing, but, beyond that, subscribers get
immediate access to all site content and access to a number of extra
site features. Please sign up today!

The adoption of open-source software in governments has had its ups and
downs. While open source seems like a “no-brainer”, it turns out that
governments can be surprisingly resistant to using FOSS for a variety of
reasons. Federico González Waite spoke in the Open Government track at SCALE 22x in Pasadena,
California to recount his experiences
working with and for the Mexican government. He led multiple projects
to switch away from proprietary, often predatory, software companies with
some success—and failure.

González Waite began by noting that he is a Mexican/Kiwi (“there’s not
many of us“, he said with a grin), who spent nine or ten years in
high-level roles in the Mexican government “promoting open-source
implementations“. Among other things, he was the CTO for the Ministry
of Foreign Affairs; “I am actually responsible for Mexicans having an
electronic passport today.” That was one of the projects that he led
and part of it was done with open-source software, which is something that
people find to be amazing, González Waite said. He served in the office for
national strategy under President Andrés Manuel López Obrador, eventually
moving into the CEO role at the National Research and Innovation Center for
Mexico.

In all of his roles, he advocated using open source within the government;
it is never easy to be a “change-maker” of that sort, he said. He left the
government at the recent change of presidents (to Claudia Scheinbaum) and
is now working on “helping people do their own transformations” to
open source, while keeping an eye out for the next big thing. He noted
that he would take questions at the end and he might not be able to answer
them all due to confidentiality responsibilities, but he is “no longer a
public official, so that gives me a lot more leeway to talk freely“.

Why?

“So why open source for the government?” One reason is to cut
costs; Mexico is a financially small, developing country that is always
looking to reduce costs, he said. Paying for licensing was costing a lot
of money that could be used to do other things. The López Obrador
administration passed an austerity plan into law, so that various
actions (e.g. government officials traveling out of the country for work)
needed presidential-level approval. That also affected purchasing licenses
and the plan pushed the use of open source.

Another major reason is to work toward Mexico having “IT sovereignty”. One
major problem that the country has had is that its IT leadership is not
technical, which means that people were signing off on projects and
licenses without understanding what they were buying. “They really
didn’t know if they are were getting good value for [the] money“.

There is a need to build up the talent within the government to support
open-source software; “there’s no point in building a whole
infrastructure on open source if you don’t have the talent to keep it
up“. There has been a lot of effort to bring in new people and to
train existing workers to that end. One goal is to go from being an
IT-consumer nation to being an IT-producing nation. It is surprising to many
who think of Mexico as a manufacturing and assembly nation that the
country does actually produce a lot of technology, he said. Much of what
was being produced within the government, though, was being siphoned off by
the private sector; more recently, that is changing, so that the government
can package and even sell its IT development.

Another goal is for Mexico to become more self-sufficient, so that it is
not locked-in by vendors of various sorts. In his most recent role, he was
able to see what was happening all across the government. One common
thread is that when agencies were asked why they were spending so much on a
particular service, they claimed they had no choice, even though there are
lots of other companies offering the same services. It turned out that
various contracted companies had corruptly put the software licenses they
bought for the government into
their own names, leading to a lock-in
for their services. Moving to open source can break those and other kinds
of locks.

There are multiple agencies within the government to handle various pieces
of the technology puzzle, González Waite said. He headed INFOTEC, which is the IT service
provider to the rest of the government. It managed the largest
telecommunications project in the world at that time to place 6,000 BTS
antennas, while deploying over
30,000km of fiber-optic cable, in various places throughout the country.
That was part of the CFE
Internet Para Todos project, which is aimed at the democratization of
access to the internet; prior to that, a large region in Mexico had no internet
because it was not commercially viable.

All of the technology projects within the federal government require
approval from the president’s office; that has been part of the country’s
laws for quite a while, González Waite said, but had not been enforced.
Under the López Obrador administration, reviews of those projects often
found that they could use open source, rather than the expensive
proprietary solution being proposed, so the projects needed to switch.

Using open-source software has been enshrined in Mexican law since 2021,
when it was specifically authorized for government agencies.
González Waite went briefly through a few pieces of legislation to show that the
government is serious about cutting costs and gaining control of its
infrastructure through the use of open-source software.

Foreign affairs

He led a project for the foreign affairs agency; it started with
separating the
data that was being stored so that some of it was put into the cloud, while the
most sensitive data was still stored in the government’s data center in
Mexico. The agency was resistant to moving any of its data into the cloud,
because it was supposedly all “national security data”, but there was no
data classification that specified which data was actually sensitive. When
pressed, the agency claimed that anything ever produced in or by a
consulate qualified; he pointed out that meant that consulates sending party invitations
and holiday greetings were violations of the national security act.
That led to an effort to classify the data and, thus, to foreign affairs
being the second government agency to store data in the cloud.

Another part of the project was to move away from Oracle and to PostgreSQL.
That change led to various threats and intimidation from the company when
it learned of the change, González Waite said. “They told me
that the entire passport system of the country was going to fall down”
and that it would be his fault that Mexico could not let anyone into or out
of the country. “Guess what? That didn’t happen.”

It turned out that the employees working on the database had heard of
PostgreSQL, but never used it. So his team reached out to the Mexican
PostgreSQL community for assistance and advice. It took around three
months to migrate the information out of Oracle into
PostgreSQL. The team took advantage of the shift to restructure the
database “because we found that our storage provider was being a little
bit naughty“, storing the data three or four times in order to charge
more money.
“Mexico is still a very corrupt country, like a lot of the countries in
the world“, he said. When you make decisions that take money out of
people’s pockets, they “start getting really nasty“.

The “biggest game-changer” from the switch was that the agency now
had direct access to its data and was not reliant on partners and
contractors to provide applications and reports. That led to the
ability to develop an electronic passport for Mexico. The platform for
that was developed in-house, “so we are not paying any licensing“;
various open-source software libraries were used in the platform, including for handling
biometric information, he said.

Education

In addition to the need for retraining technical staff on open-source
topics, his team discovered that students at universities and technical
schools were not getting exposed to open-source software either. INFOTEC started a large
internship program to place students into government agencies, but found
that the students were unable to perform their duties due to a lack of
training. That led to creating three-month boot camps for software
development and cybersecurity for the interns. The interns were getting
paid for that time, which made the training expensive for the agencies, but
it brought the interns
up to speed so that they could be productive going forward.

Along the way, he visited the communication lab at a technical university
and saw that all of the equipment there was from Cisco. The students there
did not know how to, say, configure a firewall for any other type of
equipment or using open-source software tools—and they were fearful about
learning anything else. This is, González Waite said, part of the strategy
of the companies, “where they were pretty much giving away the equipment
to the universities, but creating a handicap for the students“.

To combat that, INFOTEC developed four online educational programs, adding
cloud computing and data center operations to the two boot camp subjects,
all based on open-source software. Those were two-year courses that
included practical work within the government at various levels (local,
state, or federal). There was also a need for a “fast track”, so the two
boot camps were translated into 22-week training courses for those
subjects. These open-source education projects reached across borders, he
said, which was “a very cool thing“; El Salvador has adopted them
and more than 5,000 people have enrolled. Beyond that, the cybersecurity
course has been modified to target non-technical government employees in
Mexico and more than 3,000 have taken advantage of it.

A different form of education came about in a project to use Mifos for banks in Mexico. Mifos is an
open-source platform for handling financial transactions of various kinds.
One of the big barriers to getting the project off the ground in Mexico was
in educating the regulatory body (National Commission on Banking) about
open-source software. His team spent around six months in meetings with
the commission to do that education. That paid off and, as the project gained
momentum, the regulators had the context needed, which smoothed the
path.
The project partnered with the Mifos community; some members of the
community traveled to Mexico to meet with the regulatory bodies as well.

The other big challenge from this project was in being able return code
that was developed for Mexican banking to the Mifos project. Banking
institutions have security concerns about releasing code, but the Mifos
community wanted to add the code to its repository and tout the deployment,
which was one of the largest ever. That was a conversation that played out
over three years and eventually resulted in a complicated scheme where
older versions of the code could be added to Mifos after two subsequent releases
were rolled out to production.

The banking project had gotten its start in part because of a study showing
the enormous amount of money being spent to run the existing infrastructure
for handling certain types of loans. But, of the two different deployments
that came out of the project, the one directly targeting saving a big chunk
of that money ended up being canceled—after it had already been proven to
work. Another deployment, which basically took the code from the first and
reworked it to handle different kinds of loans, was smaller in size, but
was switched over to successfully.

The main difference is that the successful project was done for a recently
restructured agency that lacked the bias and entrenched interests that the
other agency had. The leadership in that organization was better, he said, and also
a bit desperate to fairly rapidly show the president’s office that it was
doing its job effectively.

Lessons

The regulations around open source helped a lot, “because it gave a
legal framework to explain to people why we were doing stuff“, but it
was not enough. Whenever the idea of using open-source software was raised, there was fear
because of a lack of knowledge about it, but, because the decision-makers
were government officials, there was legal fear on top of that. “That
liability could get you into jail, so don’t change the technology, leave
whatever there is, pay those millions, and nothing will happen, you will be
fine.”

Technology is often seen as the problem, he said, but he generally
found that the problems were due to using obsolete technology and a lack
of knowledge about the data being handled. There is often no documentation
of the data and its structure, coupled with no understanding of that by the
people in charge of it. Poor leadership in the agencies is another
barrier; there needs to be a champion for a change of this sort, who
understands what needs to be done and properly assigns people to work on
it. Many Mexican government officials are political appointees who do not
want to get involved in managing a project, so they hand it off to someone
else who is already doing too many other things.

Another reason that projects would fail was because the leadership decided
that switching to open source meant that no money would be needed in the
future. But servers, developers, system administrators, and so on still require
a budget if the project is going to succeed. Beyond that, some of the
government employees did not have any interest in acquiring the skills
needed to switch to open-source software. Many were happy to hand the whole job off to a
vendor due to lack of knowledge or “other personal motives“.

The biggest lesson that he learned was that projects for switching to
open-source software solutions was that either “you win big or you lose
big“. He and his team never encountered a technological or regulatory
problem that could not be fixed; both technology and regulations can be
rewritten, “it just takes time and it takes effort“. They were able
to anticipate most of the loss scenarios in advance because they saw that
logical arguments were not prevailing; when that happens, “there’s only
one explanation and that’s an alternative motive“. His “number one
recommendation” is to ensure, even before the project gets started,
that it has the right champion and backing inside the agency or
organization; that is the real determiner for whether a project will
succeed or fail.

In answer to a question about license audits (to enforce compliance with
proprietary-software terms) that was asked by a Mexican citizen who used to
work at INFOTEC as a contractor, González Waite said that all of the large
proprietary software companies “are big bullies“. He has been
called into the US embassy and been threatened because Mexico was using
technology that was not from the US; those threats were dialed back when he
explained that the government also used software and services from Amazon,
Google, and Microsoft. Various companies use the US government to bully
other countries, but they also use license audits as a reaction to projects
that move to open-source software. Every time a successful switch
happened, “six months later there was an audit“; having the right
legal team helps defend against those tactics, he said.

Earlier, González Waite had noted that the new presidential administration
had not specified the use of open-source software in any of the legislation
that has been enacted so far. Another Mexican asked him whether he was
concerned that the new administration was turning away from open
source. “I think it’s too early to know that“, he said. For now,
the relevant legislation from the previous administration has not been
undone, but that will come sometime soon, and he is hopeful that the
open-source efforts will continue.

The final question that he took was about handling obsolescence in the
libraries and other dependencies within government systems; many of those
projects will run for decades, but the dependencies may not be maintained
for that long. González Waite agreed that it is a major problem; many governments are
run by politicians with a short attention span. The politicians are
looking for recognition, which leads to votes, and then they move onto the
next thing. In Mexico, it is particularly problematic because of the
presidential change every six years; most contracts and plans do not
stretch past that transition point. It is important to address that
problem, because
technology is changing so quickly, though it would take major legislation
to do so in Mexico—for now, that problem remains unsolved.

A video of just the talk should appear soon on SCALE’s YouTube
page, but the full set of talks from the room is available
with this talk as the first
of the day.

[I would like to thank the Linux Foundation, LWN’s travel sponsor, for
funding to travel to Pasadena for SCALE.]

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!