Opportunities and challenges of state-financed Open Educational Resources . The Norwegian model – a way to more inclusion ?

Dieses Dokument steht unter folgender Creative Commons-Lizenz: http://creativecommons.org/licenses/by/4.0/deed.de Sie dürfen das Werk bzw. den Inhalt vervielfältigen, verbreiten und öffentlich zugänglich machen sowie Abwandlungen und Bearbeitungen des Werkes bzw. Inhaltes anfertigen, solange Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen. This document is published under following Creative Commons-License: http://creativecommons.org/licenses/by/4.0/deed.en You may copy, distribute and render this document accessible, make adaptations of this work or its contents accessible to the public as long as you attribute the work in the manner specified by the author or licensor.


Preliminary remarks and terminology
A: That was the title of the talk I gave: "A sustainable large-scale model for OER" and that was kind of what is most interesting for other countries as well because there are many initiatives around OER in the world but not all of them are large-scale and not all of them has been running for a long time and show proof of being a sustainable model. That is kind of in a nutshell why we are interesting for someone else.
Interview 1, 3-3 The above quote describes why the Nasjonal Digital Laeringsarena (NDLA) from Norway is interesting as an example for other countries. No other country in the world has experience with Open Educational Resources (OER) with a comparable budget and time frame.
In the German discussion on open educational resources, Norway has also been frequently mentioned as an example 1 through contributions at OER conferences and individual publications (Deutscher Bildungsserver, 2016). However, many details remained unclear. The aim of this study and the associated publication is to give readers an insight into the developments and experiences of Norwegian colleagues. Nine NDLA employees are interviewed as well as the cooperation partners, a representative from the area of education policy and a representative of a schoolbook publishing house.
The topic of OER presents the opportunity to jointly address two current challenges; namely, the usage of ICT in school and classroom teaching as well as inclusion (Müller, 2016), and to focus on the question: "How can teachers be supported in dealing with heterogeneity"? Here, drawinging on Andreas Hinz (2002), the understanding of inclusion reaches beyond the dimensions of impairments and takes into account manifold cultural, linguistic and social backgrounds as well as gender issues and sexual orientation. The two challenges are both enormously complex and wide-ranging in their own rights.
Therefore, state financing from the budget that up to now funds the publishers of schoolbooks (Muuß-Merholz & Schaumburg, 2014, p.22) is probably the most realistic way to achieve sustainable implementation. NDLA can serve as a good example, even though it does not represent the priority area of inclusion so offensively. However, the more detailed analysis of the interviews shows that NDLA is exemplary in many respects with regard to the various dimensions of difference.
Following the UNESCO, "Open Educational Resources (OER) are any type of educational materials that are in the public domain or introduced with an open licence. The nature of these open materials means that anyone can legally and freely copy, use, adapt and reshare them. OERs range from textbooks to curricula, syllabi, lecture notes, assignments, tests, projects, audio, video and animation." In the sense of accessibility for and following the example of NDLA, the following publication, unlike Kreutzer (2013, p.12) for example, also attaches importance to free availability without any barriers.
In addition to grassroots initiatives (Blees, Cohen & Massar, 2013, p.15), institutions in the federal states of Germany that developed in recent years, e.g. in Thuringia (https://www.schulportal-thueringen.de/media/), Berlin or Hamburg (https://www.digitallearninglab. de/)) or are in the process of doing so, can also benefit from the extensive experience gained in Norway.

The advantages of OER
Before discussing the framework conditions and the specific experiences from Norway, a brief outline will be given of what the interviewees consider to be the central advantages of OER.
A central approach of NDLA leads all efforts back to the question what is "best for learning", in contrast to the question what is "best for business". In this context, it was emphasized that the idea of copyright is diametrically opposed to the idea of learning and only serves to secure the business model of the publishers.
As a consequence, NDLA and its cooperation partners recommend to waiver classical copyright and the sole economic exploitation of ideas and instead to offer other people the chance to build on such ideas. This applies in particular to digital material, where the creation of a digital copy does not incur any additional costs, unlike analogue printing, for example. It is undisputed that this calls traditional value chains into question and thus interferes with publishers' previous business models. Notwithstanding, the core task of policy makers in education is the provision of optimal support for pupils and teachers -and not the preservation of business models and value chains.
Teachers in Norway are able to pursue this concept of `building on the ideas of others´, but also other countries can build on the software and content of NDLA. Free licence models are also required to enable further development based on the Norwegian software. Free interfaces (API) for the content are a beginning, but ultimately they remain insufficient.
One aspect resulting from the free licensing of content is the cross-national, joint creation and further development of content, which has only become possible in recent years due to the worldwide web.
So, that is interesting also with Google translate, we're millions of people all around the world contributing and it makes it better year by year. So that's also in a way, you know, sharing and increasing equality and a big partnership. That's very fascinating because once we used to live in cottages in wood and talked to 10 people that lived around us. We never went abroad. And now we're sort of, we're intertwined and we are developing and creating networks. This is in a way just like you create a city where people get together and you also create resources where people share ideas and ideas come together. It's in a way the same thing that you create this huge common… what should I say… conscience or awareness or space… Interview 12, 142-142 Just as the construction of materials is becoming increasingly internationalized, its use is also possible worldwide. NDLA has designed this specifically without login and password. This makes a contribution to the democratisation of learning processes, regardless of location and age, as the worldwide spread of smartphones and other access options continues to increase. The NDLA team is also aware of this aspect and wants to make a conscious contribution to it.
But we do something we believe in, I can only speak for myself, trying to go international now just the idealism in the fact that you actually want education for everyone. You want all kids to learn, that´s what I like the most. I have been to South-Africa and African countries, I have been to Asia, I have been to a lot of places where they have their cellphone or their smartphone and that´s about it. Just to bring education to everyone, I think that´s a great idea. Interview 11, Since OER are in the case of NDLA always digital, the advantages of digital media also apply. This includes the integration of multimedia content such as videos, sound files, interactive maps, graphics and task formats. Digital content from publishers can also benefit from these possibilities.
Other benefits remain restricted to OER, such as greater flexibility, accessibility for all and adaptability to students' needs.
While the schoolbooks in Norway are updated every ten years in accordance with the curricula, OER can be developed on an ongoing basis.
The best for learning is not necessarily the best for business.
When we started, we started with subjects that were many students, like languages, Norwegian, mathematics, science and so on, and we are still developing in new subjects, because it's a huge amount of different subjects in our schools and the Norwegian school system is in changing approximately every 10th year, we have a new curriculum and new plans, and we have to approve new books or digital items, so we hope to cope with that type of development, so we can cope and not produce and then we are finished and not grow old and then we have to wait for a new book, it will be a continuing development, I hope.
Interview 9, 8-8 The faster development of students' everyday reality also calls for educational materials that can be updated more quickly.
The costs of OER differ from those of digital materials from publishing houses, in that they are independent of the number of students seeking access.
If you buy from NDLA or from a publisher, it's same thing for us, but when we can use the NDLA for one, two, three or four years for the same price, if we're going to use the from one of the publisher, we have to pay for each year and there are counties with more pupils and then we have to pay more and we will have a lack of subjects in the other end. So it's about a system for getting as much as we can for our money, you know, in an ordinary way, I think.
Interview 9,  Not all the advantages mentioned apply equally to all OERs at this stage. Nonetheless, all OER providers can design their offerings according to the NDLA model in the sense of the greatest possible openness.
When I've been looking at other sites or open educational sites, it very often comes down to some kind of login or download and print. I've been looking at MOOCS for example, a lot of them are, you know, 'print this pdf ' or ‚copy this into that' and ‚watch the video here and ask the questions here and print them and email them'. They are very simple in a way, or a very sort of still in the old school. So, I haven't found another site like NDLA which is so sort of open, just click and read and… very often it is some kind of you need to download or you need to register and you need to do this and that. And then you have a site that offers 'okay, we have 10.000 resources' and then you go and look and all it is, is links, links to other sites and some of them are still up and some of them are down and some of them had changed contents. So, I haven't found… NDLA is a set of menus and you know you find it there. I know we have some issues but basically, it's all there and we don't oversell it, we undersell it if any. Searchable, lots of subjects, open, free, no hidden text, no hidden agendas. Interview 12, OER can be updated quicker than classic schoolbooks, which in Norway follow the ten-year rhythm of curriculum development.

The framework conditions in Norway
The interviews showed that numerous framework conditions in Norway were unfavourable to the emergence of a large, state-financed platform such as NDLA. How it was possible to succeed is therefore all the more interesting.
In Norway, there is less need to save money than in other countries.
It wasn't the economy that was the driving idea to establish NDLA, it was the quality of the learning resources.
Interview 9, 62-62 The number of pupils in Norway is much smaller (200,000) than in other countries, so the savings potential is much lower than in Germany or France, for example. The number of pupils in Norway corresponds to that of a large federal state in Germany (e.g. Lower Saxony).
The market for schoolbooks in Norway is dominated by four strong publishers (Cappelen Damm, Aschehoug, Gyldendal and Fagbokforlaget). They distribute specialised educational works covering a subject over multiple school years, each of which has a very high market share.
Since the pupils write their solutions to the tasks in the books, they can only be used once. The profits from such publishing are correspondingly high.
The teacher unions oppose the development of OER for various reasons. In addition to the fundamental rejection of any change by conservative interest groups, there was a discussion about the additional remuneration of teachers for their work if, for example, the school administration were to oblige them to make the materials produced by the teachers during their working hours available under a free licence. Criticism often refers to potential negative developments as opposed to actual negative developments. Another aspect is the involvement of teachers in the production of schoolbooks. The teachers involved receive royalties from the textbook publishers and are therefore opposed to NDLA because of potential loss of earnings.
In addition to these hindering framework conditions, there are others that are helpful for understanding the Norwegian development..
Ten years ago, an initiative was launched to equip schools with computers and wifi, so that each pupil now has a laptop with internet access. The teachers, therefore, had a period of ten years to adapt their lessons in this respect. All teachers also have a central login system via their schools. Other platforms can build on this identification.
The curriculum, which is binding throughout the country, simplifies the development of digital materials because it opens up more latitude for materials that lead to comparable competence developments than mandatory schoolbooks or teaching materials.
There are 19 counties in Norway. Oslo has a special role because it is both a county as well as a municipality. The counties are responsible for upper secondary education, the municipalities for primary and lower secondary education.
In addition, there are two common written languages in Norway (Bokmål and Nynorsk), so that materials must be available in both languages.
In spite of the hindering conditions in Norway, it has been possible to build up a comprehensive, professional offer of large scale OER for secondary education.

The political perspective
NDLA's failure to engage itself at the political level has the consequence that when Norwegian politicians are approached on the topic of educational materials, they focus on the needs of publishers and not on those of students. The politicians are not concerned with reusability, the possibility of further development or the question of free access. In return, they are under constant pressure from publishers.
In dialogues with publishers, politicians often shift the responsibility for NDLA to the county level. This happens in ignorance of the matter and leads to statements such as "OER stands against innovation". It is therefore a central task to make clear that the majority of financial resources (94%) will continue to be available to the market or in part create a new market.
Lobbying by publishers also plays a role at the county level and led, for example, to the Akerhus region withdrawing from NDLA for one year and NDLA losing 12% of its funding.
So what happened was, that they payed money to the schools directly and said 'You should buy learning material, digital learning material for this sum of money.' And the county said 'Oh, you shouldn't use NDLA because we have pulled out.' And they said 'Ok, in a year's time we will ask you how it went.' So politicians asked schools how it went. First thing they said: they hadn't used the money, they had used ten percent of the money. Ok, strange. Next, are you using NDLA, and they saw on the traffic, it had increased by 270% in one year. And that is because they said 'Don't use NDLA!' (chuckling), then they will use NDLA, so (laugh ing), so nobody else and they just moved in again and they didn't have to pay for the year, but they were outside, so it's like a club. Interview 1, 145-145 Although such a withdrawal may seem helpful to demand, there are currently no other efforts by other counties to withdraw from NDLA.
The newspaper publishers are -understandably -also on the side of the publishers and position themselves clearly against NDLA.
Nevertheless, it is annoying for those involved when the politicians responsible at the county level do not have the courage to stand up for NDLA and give them support.
Communicating with education policy makers is essential so that they have the students' needs in mind and not just publishers' needs.
The Norwegian Publishers fight against OER and try to influence policy makers.
Since the counties are responsible this proves to be more difficult than on a national level.

Funding and its legal basis
According to the law, the 19 counties of Norway are responsible for providing the 200,000 upper-secondary students with free analogue and digital materials. This is the result of a legislative amendment adopted in 2006. The text of the law stipulates that pupils should have access to high-quality educational material in digital and analogue form.
"Opplaeringa i offentleg vidaregåande skole eller i laerebedrift er gratis. Fylkeskommunen har ansvaret for å halde elevane med nødvendige trykte og digitale laeremiddel og digitalt utstyr." (https://lovdata.no/dokument/NL/ lov/1998-07-17-61#KAPITTEL_3) Translated, this quote says that learning in a state school or in a company that provides vocational training is free of charge. The county is responsible for providing pupils with the necessary printed and digital learning materials and digital equipment.
This change in the law opened up the latitude to create NDLA.
So one of the messages from me to all of the people, that are wondering: how could we make something like this happen, I always said 'if you are lucky enough to achieve structural shifts you should run fast and in the right direction, because you have a vacuum and possibility to do something which doesn't happen otherwise. A bureaucracy in a government, in the counties, you very seldom get yelled at, if you do the same job you did last year, so there is small incentive to do changes. But if the law changes, the structure changes, if a lot of money falls in your hand, they didn't fall in your hand last year, you have a responsibility to do something about that money. Interview 1,  This change in the law meant that parents were no longer responsible for purchasing books and materials. However, the counties resisted the possible temptation to cut costs by introducing standardized uniform books/materials for all schools. This leads to a wide range of educational materials, of which NDLA is a part.
The counties receive money from the state to provide materials and are then responsible for implementation. Almost all of the 19 counties -with the exception of Oslo (where the schoolbook publishers are located)are involved in financing the platform and its content. The main users -with increasing traffic -are the schools in Oslo, which benefit from the materials without participating in their creation.
The participating counties spend 20% of their schoolbook budget on NDLA ($50 per student). In 2016, this amounted to 2.6 million US dollars. NDLA currently provides materials for 80 subjects. The amount per subject and student is therefore less than one dollar.
The publishers brought an action before the EFTA Court against this form of financing NDLA by the state (with jurisdiction for the European Free Trade Area together with the ECJ) and lost it (EFTA Surveillance Authority, 2011). As a consequence, other EU states as well as Iceland and Liechtenstein can follow the Norwegian model in the conception of their laws and avoid lengthy legal proceedings.
The remaining 80% of the schoolbook budget is still used by schools for schoolbooks and materials. There is no effort on the part of the counties to increase the share of NDLA and, unlike the publishers claim, there is no intention to replace all printed materials.
The schoolbook publishers have failed with their complaints before the EFTA court: the funding and preparation of OER cannot be considered as inadmissible state funding.
20% of the resources for schoolbooks in secondary education in Norway is used to create OER.
The use of funds can be broken down into content and organisation. From an organisational point of view, 30% of the funds are used to pay teachers seconded from schools to NDLA and 70% are returned to the market to buy services from outside. This means that 94% of the money for schoolbooks is still available to the market.
In respect of its different tasks, the budget must be balanced between the development of new content, the development of the new platform (see page 48) and the revision of old content (see page 52).
Although the focus of financial resources on the development of the new platform is seen as a challenge in part, it is supported by all interviewees because they understand the necessity. At the same time, there is a desire to use more resources for content development in the coming years. This leads partly to the fact that the interviewees consciously want to refrain from the development of certain functionalities (e.g. functions of a learning management system), which they would very well want to use if there were unlimited means.
NDLA returns part of the money to the counties to pay the seconded teachers. The participation of the counties in the development varies depending on the concrete projects.
Funds must also be used for the professionalisation of the organisation (e.g. examination of legally sound, Europewide tenders).
Most of the money from NDLA, about 70%, is returned to the market via public tenders. This means that 94% of the money for schoolbooks remains available to the market.

The market
NDLA's relationship to the market plays a special role in the interviews. In addition to the question of the acquisition of content on the market, the main issue here is the conflict with schoolbook publishers.
The aim of NDLA is not to create content itself, but to provide students with high-quality materials.
As shown above, 70% of the available funds are returned to the market. This is done through thematic public tenders.
One condition of the invitation to tender is the criterion of openness in the sense of a free licence.
But we will say that 'what is best for learning?' Well actually openness-equality. Openness-equality that means that we, when we buy something, that we are obligated to say, that what has a quality for us, we once pay for it. Could we pay for quality? Yes of course, we could pay for quality! Well, what is the problem? Just do it! Well, you will piss off the publishers. We don't care! The market is the market and we have an obligation to the students, we have an obligation to ourselves. Interview 1, 21-21 The acquisition of content under cc licence is a challenge for NDLA.
The initial situation was that NDLA wanted to acquire content from the publishers in order to make it available on its own platform.
Interestingly enough, the schoolbook publishers who are members of "Forlegger-foreningen" completely refused to cooperate with NDLA due to the question of licences. Interestingly enough, the schoolbook publishers who are members of "Forlegger-foreningen" completely refused to cooperate with NDLA due to the question of licences.
A: Yeah, let me put it this way: If you're selling cars and you don't want to sell the cars that the customers want to buy, how long will you live in that market? It's a cheap point, but I think because if we are buying things from the publisher to put in our platform, we aren't stealing from them, so we want them to cooperate in a model, not because of we are one bit of the market, it's because we want our students to have opportunities to choose from a lot of systems. If the publisher got it the way they want, you have to buy each of them for, just for an amount of time.
Interview 9, 14-14 This refusal by schoolbook publishers appears interesting in the context of economic models of supply and demand. Instead of naming a price that does justice to the production effort, participation in the market is refused. At the same time, this refusal creates an alternative, new market.
This new market includes start-ups and smaller companies as well as the state broadcaster or the University of Bergen.
The Norwegian schoolbook publishers refuse to participate in the OER market. The desire on the part of NDLA to cooperate with the publishers is not fulfilled.
The startup Cerpus in Lofoten, for example, grew from three to 25 employees, due to a close cooperation with NDLA, because they knew how to combine technological competence with pedagogical understanding. The experience gained is now being used to drive their own projects forward. To this end, companies can make use of the interfaces (APIs) developed by NDLA and also use the NDLA database free of charge themselves. This is possible because NDLA places great emphasis on making high quality learning content accessible rather than insisting on being the only provider of the content.
However, the cooperation with smaller companies also produced unintended effects. Smaller companies quickly became dependent on NDLA because NDLA had bought everything they could produce over a longer period of time. When there was a phase in which NDLA bought less, the search for other clients proved to be difficult, as the schoolbook publishers, for example, refused further cooperation if they knew about the previous cooperation with NDLA.
The smaller providers also had reservations about the Creative-Commons licence. Here a clear naming of the authors (and maybe contact details) can help them to acquire new orders.
Overall, the interesting question of the further development of sustainable business models around OER remains. It is also important to be cautious and not to leave the business with the user data of the students or teachers to individual providers. Equally critical must be the handling of materials financed by third parties. For example, a cookbook for schoolchildren was reported, which was refinanced by the Norwegian Meat Association (matprat). Here, it would have to be clarified to what extent financing by business, politics or lobbies would have to be stated on the publication.
An interesting model of the "Kommune Forlaget" picks up the licence structure of NDLA. It used NDLA's 10-year-old texts and images, which are under a free licence, revised them and created a book version of the material, which is now being sold to schools. The reworked content was sold back to NDLA under a free licence. This results in a cyclical revision model from which both sides benefit.
Legally sound invitations to tender are more complicated with regard to the description of requirements for conceptual, creative work than for ordering computers or chairs. Since teachers generally have no experience in this area, it is important to mediate between the requirements of the tendering process and the pedagogical requirements. Thus, criteria have to be defined and a selection made based on these criteria and the price. This all has to be recorded and contracts have to be worked out.
Schoolbook publishers complain about a low willingness to pay for digital content. Therefore, a large part of the content is offered free of charge. However, the price of the offers is similar to that of paper versions. Alternative pricing models such as the sale of smartphone apps (100 sales for 2 € instead of 10 sales for 20 €) are not used. Bundles of online and offline materials are also only being discussed.
So far, there are four platforms for digital schoolbooks operated in parallel by the publishers. The task of the state, from the publishers view, is to provide technical infrastructure in the sense of an identity provider or as funders. The Danish model is an example, where the state assumes 50% of the costs for digital content when schools purchase materials from publishers. Using an identity The revision or update of OER content can also be ensured via the market.
Legally compliant calls for tender require professionalization. management system, schoolbooks can be assigned and billed to individual pupils across publishing houses.
Also mentioned is the Belgian model "KlasCement", whose materials partly refer to schoolbooks and which uses OER as digital wreath products in addition to the book. This does not question the business model of schoolbook publishers, but the dependence on schoolbooks remains, and the model does not guarantee free access to educational materials, as NDLA does.

The competition
The competition between NDLA and the suppliers represented on the market was often the subject of discussion. The role of digital resources, for example, but also the question of the diversity of offers is at stake.
Publishers are concerned that NDLA could become the only provider of digital resources and thus the only source. This would deprive teachers of choice.
But when we have NDLA as our competitor, and they get 70 million a year, it's hard to compete. Especially when you also get the price on our products, when they give it for free. So that's the big challenge and that's the risk for that in… we can come to a situation where is only one alternative for digital learning materials. In the long run the publishing houses can't manage to give all the digital materials for free, when the digital is increasing. More from the development side from our side and they're using it also more and more in the schools. Interview 3, 7-7 At the same time, several members of NDLA and cooperating partners emphasize that no one wants NDLA as an all-embracing single provider or questions the ratio of 20% for NDLA and 80% for the traditional book market.
A: Yes, I see, that if it's so that NDLA is so good that we have dealt with all kind of views in one single subject, that would be dangerous, then NDLA had become the thing I don't want! I want it to be a supplement for the others and it's, I think it's a limit how much our county can cope with in all kind of things. There will be some subjects that we don't take care of, so I think we have to take serious the opinion from the publishers. Interview 9, 26-26 All those who comment on this, wish to continue to receive a variety of different offers. The state is seen as responsible in this respect. Norwegian publishers are already receiving additional support from the exemption of all books from value-added tax (25%).
Beyond that, the publishers are well positioned as they also control the distribution structures: A: It is a bit weird but that all refers to the lock in I was talking about, 'cause if you were a company, your deepest wish would be to have a monopoly. And the publishing houses in Norway, they own not only the publishing but they also own the distribution which is the companies that distribute books to the bookshops and they also own the bookshops, so they have the entire revenue stream from. And that "lock in"-kind of way of thinking about or doing business, it also is sort of propagated into the school system. Interview 10, 67-67 Even if NDLA had been implemented as a commercial enterprise, from the publishers' point of view, competition under the condition of free licences would not be conceivable.
Interesting in the context of the discussion about competition is the following broader view on sources of information and debate, which encompasses more than a textbook or a website: I: If you could wish for something for the next ten years, what would it be?
A: That, not in NDLA and not in the publisher's minds, and I stress, not in the NDLA as well, that we produce a book on paper or a subject on the internet that are trying to tell our pupils that if you read this, that you No one (even nobody at NDLA) wants NDLA to be the only content provider. Everyone is in favor of varied offers, where the teacher is responsible to choose.
have learned everything you can learn in this subject. I don't like the idea of one book coping with one subject in the school. I wish in the future, that when we are dealing with a curriculum for our students, they can find the way of learning that suits themselves, something in one book, something in another book, something in the computer, something to speak with the teacher, something to speak with the grandpa or whatever.
Because we have to realize that future good citizens and workers have to analyze, see things together, choose and be able to cooperate, so I think my wish is that we shall support another way of learning than we are trying to do today, the debate about between NDLA and the publisher is far away about a debate about an old-fashioned learning-view. So I think we have to change our way to deal with these things in the future. Interview 9, 59-60 Digital resources are not seen as a substitute for books.

Collaborative development
Due to the geographical situation in Norway and the cross-nation financing of NDLA, a very digitally oriented, spatially distributed cooperation has been necessary from the outset.
After the initial development of structures and content in subject-specific editorial teams, the development process became more open and collaborative. For the development of materials in the fields of history, religion and sociology, workshops were held with participants from all walks of society with an interest in the subject area and its communication.
These include, for example, museums, the Bureau of Statistics, and universities. Synergy effects can arise here if these institutions also use Creative Commons licences. For the institutions, this leads to a greater reach and the embedding of content in larger subject-based contexts.
But what we do is we make quality assurance on this and if many big museums have animations, they use hundreds and thousands of Norwegian kroner on each animation. And they can give it to us for free. And it is better for us to take that material contextualize it for secondary school in Norway and that will be much better resource than we could have if we used 50000 kroner on producing it ourselves. Interview 3, 49-49 Such cooperation, however, takes time, which NDLA has at its disposal as a state institution without the pressure of financial success. This cooperation with external institutions (e.g. the Office of Statistics) then enables, for example, the automatic updating of NDLA content via interfaces (APIs) of the cooperation partners.
In addition, there is a discussion on the extent to which third parties (teachers, pupils and interested external parties) can be involved in a structured way, so that, for example, specialist networks are created that are involved in the revision of content.
This also applies to the creation of content for subjects that are only taught rarely and are therefore commercially uninteresting for publishers. It is also unrealistic for NDLA to set up specialized editorial teams for all subjects of vocational training without changing the budget. Here, committed teachers can take on smaller task packages and receive a work contract for the equivalent of 3,000 €.
A: Well, you know, a lot of the subjects are not commercial, in a sense that it doesn't pay off to create a traditional book, because there are not enough students to buy them. In these subjects, teachers often come up with an idea of their own and create their own sort of learning resources. And I think that if you can get four, five, six teachers to cooperate and say okay, you can have this one, I can have that one from you…. I think they're willing to cooperate because, and again, they don't have to do all the work themselves and… the quality of work improves… and it's also interesting to learn how other teachers work. So I think they will find that it is in their own interest to cooperate, because they can't just go to the book shop and buy a new textbook. The publishers don't have a …, well they don't see the commercial potential in it. I'm talking now about a lot of subjects in, within vocational studies, and also in general studies at the second or third level. We're sort of just scraping by, you know, with resources that a book can offer. So I think that teachers will see that it's in their own interest to cooperate and participate. And of course, a web site offers so much more functionality, than a book of course. But I mean, publishers can also create websites and some of them have created great webpages, so everyone can do that. We have a system in NDLA with what we call teacher contracts and that's also a sort of a, a way to sort of Expanding and updating large-scale OER platforms require capacities that in the long term cannot be met by editorial teams alone.
deal with this. You pick teachers who show a particular interest, or who are creating or who like to write. Then you offer them a contract of let's say 30.000 Norwegian Kronars and during the year they produce learning resources. Then you agree….okay, you do some… you look at these literary tasks or you do this topic.. and then they hand in. That is also a very efficient way of working.
Interview 12, 20-20 In this context, it is important that the cooperation partners approaching NDLA plan from the beginning and not only at the end of their project, because then it becomes more difficult to integrate the contents. The cooperation with the University of Bergen, for example, was more difficult than necessary because licences had not been clarified upfront. Here a checklist or roadmap for external cooperation partners might be helpful..
Teachers and students can be involved in the creation and revision of OER content.

Challenges in everyday school life
As in other countries, the ICT competence of teachers in Norway varies widely. In the last 5 years, however, there has been a noticeable improvement.
This may be due to the generous provision of IT infrastructure to schools, which led to a qualification offensive.
A: You know, Norway is the land of milk and honey so I realize. Perhaps it's not a good start for a country to start with a deal to spend a huge amount of money to give the computers for free, I'm not sure if that's the right way but it was the way it come out. So we had spent a lot of resources to develop the competences between the teachers to deal with this way of learning, some teachers are very good and some teachers are just clinging to the book and use the computer for certain exercises and this is the way, so it's very diverse.
Interview 9, 58-58 This advanced IT competence is also evident in the applications for positions at NDLA from teachers whounlike in the past -have generally already had contact with OER and Creative Commons. However, this is particularly the case at upper secondary level.
Contact to OER and CC licences is not yet synonymous with competence in dealing with OER. NDLA has developed a platform that clearly informs employees about how to deal with OER. This could also be used (in further training) by teachers.
Cooperation with universities and practical phases in the field of teacher training is one of the areas of work in NDLA that is not yet particularly developed. Here, there may still be potential for more intensive cooperation in order to benefit from the work and experience of each other.
In addition, the question of the required digital competence on part of pupils and teachers arises. Is the consumption of multimedia content sufficient or should it not be about the creative creation of content? Accordingly, other contents of teacher training and further education are required.
But I'm also aware that the problem is that they know From the perspective of the teachers, the continuous availability of the material (in unchanged form) is of particular importance. It can be frustrating if the URL, which worked yesterday, the next day leads to a void or a changed text in the classroom. Therefore, NDLA's archive function is very helpful in providing a reliable working basis for students and teachers. It should be examined to what extent such a function could be included in an introductory video for teachers.
The question of technical availability also falls under the category of reliability. Even if the responsibility for malfunctions lies with the technical provider of the website, subsequent dissatisfaction is directed against the offering institution.
The systematic training of teachers and creating linkages to teacher education are a challenge for OER platforms.
Of course, we talked about what to do if you should start a similar initiative and that is, I think… you make sure that the editor, the production system is up and running and bug free. And you make sure that you cooperate with schools that have the necessary facilities.
Because if the network is very bad and if the access to the computers are not sufficient it will reflect badly on your initiative. So, it's better to grow in a way that things work because the teachers don't… if the provider of the web server or the network provider hasn't done its job properly and your content is not showing up properly, you can make sure that the teacher… , they blame you. So, you need to make sure it will work in the classroom, you can't say 'okay, on that day, the network provider didn't provide what they had promised us, it doesn't work. You get the blame. Interview 12, 148-148 The temporal continuity of OER content through an archive function creates security for the teachers.

Openness
The openness of NDLA affects both the developed platform software and the content.
A central question that NDLA has repeatedly asked itself and is still asking is how the budget can be used to achieve goals, as sensibly as possible This includes the infrastructure for providing content as well as the content itself.
In terms of a cost-effective solution for the infrastructure, it is necessary to check whether there is reliable open source software that meets one's own needs. In this case, it would not be necessary to develop your own solution or to buy or rent additional software. Instead, one could build on this existing possibility. The experience with the open source content management system Drupal showed that an individual solution became necessary after a few years. Thus, this was newly developed according to the requirements of a state-run OER platform and made available under a free licence on Github (https://github.com/NDLANO).
The creeping death of Adobe Flash showed: if no satisfactory open solution is available, then it might make sense to develop one instead of entering a closed ecosystem, from which one can only find the way out with pain. How NDLA reacted to Steve Jobs rejection of Flash on the iPad will be discussed in more detail in the following chapter.
As far as content is concerned, NDLA is prepared to pay a reasonable price for the creation of high-quality content. The openness of the content is (as far as possible) a non-negotiable criterion, which results from the requirement of "high quality". If materials are not subject to copyright, students can deal with them creatively in a completely different way.
It's better for the students to be able to share it, to reproduce it to be in a dialogue with their own learning material and also they are more critical, when they use open learning material, because they have to ask themselves who has written these. It's not just the publisher but that is something else, it is a person. Interview 1, 21-21 Openness requires transparency of sources and a critical examination of the authors and the information. This desire for critical discourse and the joint further development of the materials is also the reason why "Arena" as a place of discussion is part of the name of NDLA.
Direct, unrestricted access to all learning content is called for and implemented radically by NDLA: In the future, do preach about openness! That's the most important thing. No barriers, no login, no boundaries and no anything. You're free to use it. Interview 5, The ideas about open licences on the software side are shared by some, but not all, cooperation partners. Thanks to Steve Jobs, openness at NDLA is not just limited to the content of OER, but also includes the software and formats of delivery.

Open formats -the example flash and h5p
In education, the software h5p was developed with the support of NDLA as an open standard and as an alternative to Adobe Flash.
The starting point for the development of h5p was Steve Jobs' decision not to support Adobe Flash on the iPad. This turned out to be a big problem for the providers of multimedia materials in the education sector, as their content was no longer accessible on a widespread platform. With the open format, other educational institutions have also adopted h5p and commissioned Joubel to create more content types according to their own needs. Meanwhile, the development is supported by 25 organisations from 10 different countries (e.g. Victoria University from Australia). NDLA and all other users benefit from these new modules.
Due to the open format, translations are much easier, so that OER platforms can more easily recreate multimedia content in their own language.
Things we sold to NDLA, we sold the e-lectures to them and I think the price went, it was over half a million kroner for one hour animated e-lecture. It's quite expensive so they could only afford four hours or something per subject. And obviously that's far too little to cover an entire subject in a good way. And we see the same problem. The publishers in Norway their budgets per subject is much much smaller than the NDLA's. So, they are nowhere close to afford really good content that covers the entire subject. Norway is a quite rich country, even if By financing the development of h5p, NDLA has laid the foundation for an open educational multimedia format that benefits many other educational institutions (including those outside Norway). we are small we have okay budgets, so other countries have the same problem. So, in order to get really good content for the entire subject you need reuse in a really big way. And in order to achieve that you need a standard for how the interactive content is created so that NDLA can share all that content and someone in South Korea instead of building it up all over again explaining Pythagoras via interactive technology for the ten, you know they are probably hundred thousand, times. Has it been done then they instead they just translate and do this content and they have a really good start. I hope we will get there. It looks good right now. Interview 7, 41-41 Through the integration of translation platforms, the first translation into other languages can be done faster and cheaper than if the editorial teams themselves were to do it.
Accessibility and access to smartphones and tablets are guaranteed by h5p.
In future, the content created with h5p under a Creative-Commons licence can be shared via an h5p-hub, so that content can be used more easily. Here, too, the question of metadata is raised in order to ensure that it can be found.
The previous content in Flash format is currently being elaborately transferred (partly by screencasts) into new formats.
The content types include many different formats (see Figure 1).
The frequently used content types are interesting. For example, the type Course Presentation, which resembles a PowerPoint presentation, is one of the most frequently used. At the same time, it presents the greatest difficulties to the developers, as the design varies on different end devices.
Some formats also contain voice input, whereby the development of own voice input solutions was omitted in this case and therefore was based on the solutions of Apple, Google and Microsoft.
Switching from NDLA to h5p does not mean that all users of NDLA are happy with the replacement of Flash by h5p. Particularly the freedom in design of Flash is missed. Nevertheless, everyone is aware of the potential of h5p and the need to switch to an open standard.
With h5p, Joubel aims to develop the format into a worldwide standard. The speed of the development will be of decisive importance. Is it fast enough to implement a sufficiently large number of formats and a large amount of content via h5p? There are comparable initiatives from Kineo with Adapt Learning, which, however, rely on SCORM and their own server structure instead of plug-ins. They also focus more on in-company training.

Licencing issues
Licencing issues are a central, yet complex, issue in the context of Open Educational Resources. For a general overview on this topic, the reader can refer to the remarks made by Kreutzer (2013). NDLA, too, has addressed many different aspects of licensing issues.
The free licence is seen as an indispensable working basis as it provides the fundament for students and teachers to further develop their own materials.
So, if we know that pupils learn more if they get to develop their own learning resources, then we have to find a way to present that to our users. And in that aspect we are, as open educational resources as a concept and creative commons as a licenses, is essentially for us. Because if the pupils and the teachers can't edit on our material then we probably can't offer them the best learning resources. And you couldn't do that on a copyright license, that would be illegal. That is probably the one small detail which is going to make this whole area within education disruptive. Interview 3, 7-7 NDLA is currently investing time in the verification of old licencing information. Especially in the case of older content, there are unclear licencing situations. In order to avoid such a situation and possible warnings due to licence infringements, it is necessary from the outset to implement a system for the accurate documentation of cc licences.
Depending on the combination, the use of different licences can lead to numerous traps. Therefore, efforts are being made to use cc-by or cc-by-sa for everything. These are the preferred licences because they are the most open possible. The use of cc-0 is not possible for Norwegian authors due to the Norwegian copyright law. At a minimum, the licence "naming and share-alike" (cc-by-sa) is expected as a guideline for the seconded teachers paid by the state. The licence with the sharealike regulation is also intended to encourage others to make their content available under a free licence.
So, we should have everything on same license.

I:
Are there any efforts to change licenses to get it all to one?
A: Yeah, because especially in… to get it right, we know that the one license that will be easier, but also when we should mix this together. Like I made a video and I need a picture almost, the most of our picture is non-commercial and share-alike. And then I get put together with a video I make no-derivs. But what license should be on the video? It lasts. And the picture and rest, the multilicensing part, that's difficult. Like share-alike… you can't put a no-derivs on a share-alike object.
Interview 4, 32-34 As long as the licences are not all the same, an approach used by NDLA to create transparency is to display the licences bundled at the bottom of the page to do justice to the right holders. The upcoming version of NDLA's platform contains simplified menus that help users to continue using content and gain information about the actual rights and obligations of the specific CC licence in understandable language.
So far, students have had limited awareness of copyright law and the use of CC licences.
So, for the users -they don't care! The users, they don't know that we are using open learning resources. They couldn't care less. They are not interested in OER. Why should they be? They just want to learn. Interview 1, 102-102 An exception to this are pupils who have chosen the field of media at upper secondary level.
Information about licences and their implications for further use is important for teachers as well. The issue of licences is also addressed as a barrier to the sharing of content by teachers. Based on a vague awareness of the copyright infringements committed and a lack of knowledge on how to avoid them, teachers are reluctant to share content online.
A secure documentation of OER platform licenses right from the start avoids concerns in the future.
When presenting the CC licence, another issue is the question of how relevant revisions are to be identified in a traceable and transparent manner. The question of the naming of relevant revisions is particularly important when a wiki-like model of revisions is developed. When is a contribution an addition to the content that deviates so far from the original that it should be indicated?
It is also interesting to purchase images from professional databases under a free licence. NDLA buys professional images under a free licence from ScanPix (cc-bync). This company in turn buys images from iStock. This way iStock images are freed for non-commercial use.
When creating own pictures under cc-licence, the Norwegian personal rights law represents an interesting but hypothetical challenge, since the persons depicted have the possibility to revoke their consent. However, concerns about the misuse of cc content, e.g. by extremist political parties, are unfounded due to the abuse prevention clause in the cc licence.
So far, NDLA has not received any court warnings due to licensing issues. In case of difficulties, NDLA was contacted and subsequently removed the content. The use of music under a free licence on Youtube is more time-consuming, as incorrect rights claims can occur, especially with classical music, since several orchestras have recorded the same music. When purchasing music licences for other music, the Creative Commons licence is restricted to non-commercial, no derivatives.
Similar negotiation processes with rights owners are necessary for films and texts. Film rights are one of the And also the right holder said that 'oh we need logon' and I said 'okay, but you have to sell it somewhere else because I'm not paying for it because it's logon.
Because it should be completely open without logon'. And then he said 'oh but we need something'. So this is what happens. So look, I may click on this one and it says 'You have to click on a box that say you are a pupil at a school in Hordaland county. If this is not correct click here.' Okay, I click here and then I will choose a different county, I will choose school, I will determine if I was a librarian, teacher or student. I say librarian and I will click save. I will then choose if I want to have subtitles in Bokmål and Nynorsk and I will say save and it will start play. So I haven't logged on, I just said which school I am attending.

I: Okay. So basically everyone in Norway can do that?
A: Yeah, with very many films, very many short films, animations, documentaries, TV-shows. So, we have actually a lot of German films, when I come think about it. Because we have German as subject. So we got the rightholders organization to agree to this and everything is okay. Interview 1, 83-85 Free licences are also difficult to obtain in the field of literature. Some of these are classics that become public domain after 75 years. It remains to be seen to what extent there will be more open formats on the literature market that will allow current literature to be embedded, at least in extracts.
From the point of view of cooperation partners, the dispute over licences has been an integral part of a development process. Over the course of time, it became apparent that the hoped-for resale of the contents to third parties did not work in a promising manner.
A: We were also in the beginning a little bit reluctant to their licensing wishes. So, in the beginning we didn't give them Creative Commons and we didn't give them exactly the licenses they wanted. We had board members who were quite traditional and who were hoping that we could resell the content we made for NDLA to others. And that meant don't put Creative Commons licenses on it. Luckily, there were many others as well who thought the same so they didn't get, I don't think they got Creative Commons license content in the beginning from anyone, at least not a lot.

I: And how did this change?
A: I think it was we saw that we didn't succeed in reselling it anyway so why insist on it? When they did their purchases, they had their own criteria for what kind of license you offer and I think that criteria got a bigger percentage of the total score. And we just decided that we have created stuff for NDLA for a while we are unable to resell it so it's no point in... We can just as well get some more points when we compete for the next jobs from NDLA. But I think it was pretty much like that. It took some time before their philosophy and vision that we started to believe in it. Interview 7, 53-55 NDLA is committed to providing teachers and students with understandable information about licences and their consequences for learning processes.

Sustainability
In interviews with NDLA members who had previously worked in other projects it became clear that these temporary projects were often not very sustainable.
I also get to know a lot of people around the country doing other things, very interesting projects. And nobody took care of them, they were funded for two or three years and then they died, so we were looking for a model to be able to let them live and building it together and really, NDLA is my name, I was the one who created it. Interview 13, 4-4 Sustainability was seen as one of NDLA's great potentials and was cited as one reason why free licences for software and content were urgently needed.
I think that most people in NDLA -me included-we are not, like, fighting for the brand. We are fighting for the students, we are fighting for better schools. If NDLA tomorrow would disappear we would be so proud because it would be replaced by something better. Because we have secured open license for infrastructure and on content and that makes it possible, if you unplug it, someone else could plug it in -and pay for the electricity. They could just hijack it, they could make a fork. Interview 1, 65-65 Accordingly, even if the funding should not be continued, an extension is possible without further ado.
At the level of teachers and students, this means that they have the right to copy and store content in a format that they can access for the rest of their lives and use and disseminate in other contexts.
This is not possible with the current licencing models of the schoolbook publishers. In Norway, too, time-restricted licences are provided for digital schoolbooks.
I: So, if it's epub3, you got one copy for yourself as you keep it for the rest of your life or is that also with the…?
A: No, this is license and these are different kinds of models here but mostly you have them for three years as a student. Here we have lots of (playing a video/ song). You have songs, you can write, you can mark and all that kinds of thing you can keep of course. But that's a license based product. Interview 8, 27-28 Since NDLA covers both the classical school subjects of the upper secondary level and the vocational training area, the use of NDLA materials by learners extends beyond everyday school life into universities and professional practice. This is also made possible by the absence of schoolbound logins and the associated direct searchability of content via Google.
Thanks to the cc licenses, NDLA is more future-proof than predecessor projects which disappeared after the end of the financing.

Pedagogical considerations
Pedagogical considerations loomed large in the interviews and accordingly encompass numerous subcategories, of which the central results will be presented here.
14.1 Dimensions of diversity as content From the perspective of inclusive pedagogy, dealing with diversity at different levels is particularly interesting. Which aspects of diversity are taken into account? Is diversity depicted or thematized within the materials? How is the diversity of learners and teachers taken into account? How are differentiated offers made?
So far, there are no rules for the consideration of diversity, but they are discussed and defined as additional documents on the pedagogical platform.
The Norwegian law on the approval of schoolbooks used to prescribe the consideration of gender aspects as mandatory. This was abandoned in 2001. Nevertheless, the way of thinking has been internalized by the parties involved and, in case of doubt, would be demand ed by the users.
The challenge now does not lie in understanding the contents of the various dimensions of difference additively, but in interweaving them with the object. Historically, such a predominantly additive solution already existed in connection with the social role of women.
I: Is disability part of the materials? So are there images of disabled students or is it a certain topic here or there by some short stories or tasks or something like that?
A: We've been discussing... you see some years ago when we creating just... I have to use the example about women. When we were, you say, more or less forced into, 30 years ago the government or the department said 'you have to explain how women were in that. Then always, we've gotten a little part in the text saying the woman in history and so on and so on. And that could be the thing today, it's not the ability, just have some texts about people with some difficulties. But I think that is always the wrong way to take it. We have to mix it into the whole text and the whole issue to show that those are people living together. And we have to consider their needs anytime in the day, not just in some parts. So, I think our main goal is to mix it together and show it in the whole context. Interview 5, 68-71 In addition to the consideration of women's perspectives, the inclusion of content that is interesting for different groups of pupils (including boys and girls) is also addressed. Due to rapidly changing real life, however, it is not very easy to maintain a current reference.
The dimension of sexual orientation will play a role in the current development of materials for the subjects of sociology and religion. It will be less centrally thematized in other subjects. All in all, the liberal, humanistic orientation of the NDLA colleagues also shapes the presentation of the contents.
The dimension of cultural background is a present topic within NDLA. However, the implementation is rather difficult due to the employees' own, less diverse backgrounds. Nevertheless, there is an effort to systematically incorporate different cultural backgrounds into texts and images. Analogous to the other dimensions, it is important not to make this additive. An example of the consideration of local minorities can be found in the contents about the people of the Samen, which are woven into the texts.
The dimensions of dis/abilities have only been dis cussed sporadically in the materials so far.

Differentiation
A special potential is seen in the possibility presented for differentiation with help of digital materials. The conception of differentiated offers for individual pupils is made much easier by ICT. Thus, the level of difficulty can be Gender issues in OER are addressed through a combination of awareness and user feedback at NDLA. assigned to individual learning paths and additional videos, pictures and audio documents can be used for illustration. Working with well thought out individual learning paths enables teachers to work without classifying into stigmatising academic levels (A-level, C-level).

Cross-subject learning
The use of content in different contexts and the linking of content is interesting when creating a cross-subject platform such as NDLA. This is one of the challenges in contextualizing content. Where can an object or a process (e.g. from mathematics or chemistry) be found in other contexts?
The presentation of content from different perspectives (e.g. democracy, climate change, sustainable development) is on the agenda for the development of new content in the context of the next curricula plans.
14.4 Teaching and learning research NDLA cooperates with research institutions in Norway and beyond and is oriented towards the results of teaching-learning research. The reason for this orientation is quite interesting: Because today, we focus too much on subject contents, too much on Napoleon, to little on how to remember things, how to work…, how do you become better learners.
If you take a sprinter like Usain Bold, everyone will recognize that he spends hundreds of hours on very small adjustments. He always uses the best gear, the best things money can buy. We kind of look at the students, they're also going through a sprint… how can we sort of…, how do you learn best. What's your ideal posi tion. What can we improve on. But you know, we don't have…, hardly any focus on that… It's more like… okay you got 5 out of 10… work harder. Maybe, it's working the wrong way, maybe it's not.., maybe there are tools here, that we can provide them with, help. Interview 12, 97-101 In addition, NDLA's own research is carried out in cooperation with pilot schools. The procedure is geared to the needs of the users. The users are both pupils and teachers. New materials are evaluated through observations, tests and interviews before they are produced on a larger scale.
I: So, teachers and students are your users in terms of user needs?
A: Yes. We do quantitative analysis, we ask people, we do a lot observation of them, we have five to seven different sources, where we use, I would call it a research pyramid technique to find out where to go. You can do one study and the study tells that 'well, you should do that and that and the rest is more blurry'. And then okay, let's do another study. Let's not just measure this on the logs, on the service, let's go to the classroom and ask people. And we can ask them and we can find out, not by leading the questions, but if there are things coming up there matching, the measurements we done then, well, maybe here is something we should do.
And then we could test it, because if you ask a pupil how this should be done, they will probably lie, also Interdisciplinary learning is made possible by a common OER platform, but it does not happen on its own.
Individual learning paths allow teachers to easily adapt to students while working on the same topic.
teachers, everybody is, because you have to observe what actually happens. If someone says that the first thing to do is look here, then we know, that is probably not where he is looking first because we know from eye-tracking that you will look there. So we use all this sources and when many sources tell us the same thing, then we test it. We build it and we test it. And we try to find the next step. We are so early right now, that that kind of information will give us many, many years of work from that start. Interview 3, 43-44 In the context of definitions, for example, the question is how a term can be best explained. Can this be achieved by means of a text, an image or a video?
When developing new content types, tests are carried out with comparison groups before the content is made available to teachers for use and further adaptation.
And we will also always test new types of content, always do the development and check 'okay, is this better or is this better' (A-B-testing). Until we reach or we see that 'okay, this is good enough' then we give it to the teachers and pupils and they can change it if they like and they can use it or don't. Interview 3, 49-49 It should be noted here that the interactions with the material take place against the background of a variety of framework conditions and environment variables, which, unlike in other research areas, cannot be compared with reasonable effort.
The tests with the help of screen readers give hints about the use of the contents by different students and influence the further design.
I: What would you say were the interesting out-comes or insights from the student-testing?
A: Well one of the things that I found the most interesting and that really made me think was when we used eye-tracker. How they actually read. We have a lot of you know thoughts and ideas and even opinions on how students read and to be honest they didn't read the way we thought they would read. So we had created this page. Like we had the main text and then we had some additional texts on the right hand side that we thought would be an easier version of what was in the main text ,simplify it like an abstract or like some key issues there, so we thought that that would help the students. But they didn´t even read it. They weren´t bother with you know reading stuff they just thought like: "Well if it's important, it's here like in the main text. That is just extras." So they didn´t even think about it and I know that a lot of our people have spend a lot of time thinking about how to create that part in a good way. So that it should actually help the students. Whereas they don´t care to read it. That's interesting. Also when we used the eye-tracker we saw that how we organize our texts and how we build them. We had one student struggling with reading a lot and I think he used 4 times as long as the other students to read it and how he read and how he found it easier if the paragraphs weren´t too long. The headings were described well. And were we had a lot of headings, if we used instead of full texts you had small points or small bullets. So that´s the way he actually read. How that helped him. That was also good for us because we tend to like,I mean we are teachers, we love words especially Norwegian and the language teachers they write and write and write and also the craving for visual elements, how that helps them at least the possibility to be able to: "Okay you can read but also listen or you can also watch a video and you know using multimedia learning material as well." So all those things were actually helpful for us. Yeah. Interview 11, 50-51 The feedback from the field studies saved time in the development of new content, and the design was geared more to the needs of the users.
A further approach to the consideration of user needs is the creation of situational offers that include time and place of use.
Since pupils are the users of the platform, their needs should be central to the development of OER.
But we know that when a pupil has been reading on preparing for exams for three days and has been sitting ten hours today preparing for those exams, then we know that when she is going to start at seven PM in the evening, start to prepare for small glossary test tomorrow, then we know she isn't really interested in reading more. Then we know she needs something to listen to or something like a video or something else. So, it is not only about strategy so how pupil best consumes learning for her personality it's situation based. When I'm here in the bus I don't want to read page by page of text. I want to put on my headset and see a introduction video. So it's not only learning strategy, it's situational learning.
Interview 3, 50-51 The analysis of the resulting user data can also help to decide which materials are interesting or helpful for which group of pupils. For the opportunities and threats of the BigData approaches, the reader can refer to the chapter on user data (p. 16).

Proselytisation for pedagogical approaches
In its effort to take the pedagogical freedom of teachers seriously, NDLA has decided not to carry out missionary work for certain pedagogical approaches. This may be difficult for the individual, but missionary work is still considered inadmissible.
I: So, do you try to influence the teachers in a certain way of teaching?
A: That's a hard question. Because we would very much like to do that, I think. Anyway because I think if you read that paper you will see that we are telling the teachers that if you're going to use a computer in your lessons, you have to let the pupils be active. You have to let them work, you have to let them cooperate and so on. If you let them hide behind a screen and do the teaching from above, then they will soon be on facebook. And I don't mind pupils being on facebook. I think when Facebook is an issue let's use facebook to create good teaching or good learning. But the question was 'do we influence the teachers'. I think we can't do this as much as we would like to because there is a standing word or something in Norway that the teachers should be free to use their own methods for learning and I don't think that is right to the total end or something. I think we have to admit there is some methods that are better than others in learning but we can't officially go out and say 'do that!' No, we can't do that. Interview 5, 81-82 Rather, the aim is to achieve the greatest possible degree of openness for the use in different pedagogical settings. Due to financial limitations, however, this cannot be achieved for all materials and every approach.
I: And what about the different ways, for example you said 'problem based learning', so how do you decide what pedagogic strategies you are using for teaching? So, for example 'problem based learning': Are there certain ways of teaching you prefer and you say 'well, they should be able to work in a group' for example?

A:
I have to answer that in two ways, when you talked to 'Leonhard', he will present a pedagogical platform to you and I don't know if you are talking to 'John Eide' during your visit here, but he is working more operational on that. They will say 'we know this, these are better learning resources than these, we know that from research. As long as we know anything proven by evidence based, then we can say, we will probably work on that and resources we know is not that good, we won't work on it'. So in that way we will follow the research, we will follow the evidence based practice and further try to participate in the investigation in what works and what doesn't work. On the other hand, we're not following other… I would say, we would have to make our content available for every pedagogical approach. That means, for example, we have some math subject, they are produced for a way of teaching pupils. The biggest challenge for us, in addition to moving to a new platform would be how to make that learning material available for other pedagog ical approaches. So I would say we would probably wish, that all our learning resources should be possible to use Openness at NDLA also means the openness of materials for different pedagogical approaches.
in a problem based learning approach in the same way as we would like it to be possible to use it in another pedagogical approach. Nevertheless, the introduction of digital media is a major change for teachers who maintain a classic book-and teacher-centred teaching style. This goes hand in hand with a democratisation of digital means of production and thus a multitude of new modes of expression for pupils. The possibilities for activating pupils, which are interwoven in NDLA, may also have an influence on teaching.
14.6 Promoting learning processes through disruption (information is not knowledge) An interesting question that emerged in the interviews was "What kind of latitude does a state learning platform have in the presentation of content?" Unlike Wikipedia, whose movement was an inspiration for NDLA, NDLA is not about creating exactly any one form of content representation, but about making the subject accessible to students at different levels of difficulty through different paths.
It should be considered to what extent a technically correct version of a text is necessary for the design of learning processes or whether an erroneous version might even be more useful for promoting discussion. With reference to Frank Zappa's "information is not knowledge and knowledge is not wisdom", it was discussed whether a deliberately erroneous version with a reference to errors contained in it would enable a more in-depth discussion than correct materials. This is especially true since learning processes involve more than information intake, e.g. discussion and linking with known content. For a state platform such as NDLA, however, it is unrealistic to present technically incorrect content owing to potential conflicts with publishers.
The discussion about the difference between information extraction and critical alienation on the basis of specifically erroneous, discussion-stimulating sources is an interesting suggestion in the discussion surrounding OER.
14.7 Multiple ways to access the same topic (video, text, audio) The central opportunity of digital media and OER is the possibility to offer various approaches to a topic and thus to be more than a digitized textbook.
But of course we also studied the curricula and we studied textbooks to see what can we learn from textbooks, because there is always a tradition you can learn from, even though you break with something you can always learn. I think we also had a lot of focus on creating a lot of interactivities for the students, so it shouldn't just be reading. And we focused a lot on multimedia content, so rich content. Because I think that's part of the web experience that you have a lot of media, you know, a lot of pictures, a lot of videos, lots of sounds -part of the experience.
And we also were very committed to creating a variety of resources so you should have a lot of variation and you should be able to choose from two or more sort of approaches to a topic. You could use all the textual approach or you could watch a video. So, we weren't able to do that with all the topics, but that was sort of an idea we had. I think it was a very good idea. But in retrospect, creating so much content also produced prob lems with maintenance, keeping things relevant and up to date. So I remember we had lots of texts about Obama and 'change the world' and... But, when we finished sort of creating the various contents, it was... you moved on to other topics, to English at second and third level, you know with 17-, 18-years-olds. Interview 12, 3-4 The problems with content maintenance are discussed in Chapter 19.
Provoking learning processes through targeted built-in errors in OERs is not easy with state-run platforms.
14.8 Activity of pupils The pupils' own activity in cooperation has been discussed several times and is part of the pedagogical platform document.
Anyway because I think if you read that paper you will see that we are telling the teachers that if you're going to use a computer in your lessons, you have to let the pupils be active. You have to let them work, you have to let them cooperate and so on. If you let them hide behind a screen and do the teaching from above, then they will soon be on Facebook. And I don't mind pupils being on Facebook. I think when Facebook is an issue let's use Facebook to create good teaching or good learning. Interview 5, 82-82 Student activity means both online activities and offline activities. Online activities include changing from consumer position to the role of content producer. This can be done through photos, videos, podcasts or blogs. It is not just a question of creating content, but also of critically and respectfully interacting with one's own performance and that of others. The Internet has made a considerable contribution to the democratisation of means of production and distribution channels, and at the same time requires addressing their responsible use.
[…] we want them to go away from saying 'what you are going learn is this book', which is what they're always thinking and a lot of the students and teachers, they didn't read the curriculum, they just read the book and I know teachers that they haven't discovered that there is another curriculum, not the last one and not that before, because they are doing like they did on 1994, so you know, it's a habit to deal with, they just do it like before. And the students, ok they think, they have a easy day, when the teacher is just talking, so they, the students, don't have to do anything, you know. But we want them to teach in another way, to activate the students quite another way. And also of course use digital tools and working with what the students are doing every time, they are using their phones to make videos or to listen to speak or yeah, and the students, they are living in a free world, where they can express themselves in a quite other way than when I was that age, because everyone what I read, always some other had written but now they can be writers and they can publish and they have a blog or they have a… so they are more active in society and we have to use that to learn them and to fix that role for them in a good way, to use this freedom and do best for themselves and for the society. The aspect of cooperation as an integral part of the pedagogical platform is emphasized several times and is of major significance in the context of digital media, since the danger of social isolation is quite real with continuously individualized task processing.
So, yes, so we worked out a platform for what kind of pedagogy we are going to use and so we're thinking that activity uh pupil's activity is very important, stressing that point very hard, cooperation, learning to work together is very important, that's the stressing and then you can say that we tell students or the pupils where they are all the time, so they will reflect on their own learning and so on. Interview 5, 24-24 14.9 Offline activities of students One of NDLA's concerns is the activation of students through tasks that are not only to be solved online and that require interaction with the world and with other people. This includes, for example, studies and photo documentations of ease of accessibility in one's own environment.
A: Yeah, yeah, we are trying to do both. Learn from all the digital University and working with digital learn- The democratization of production tools and distribution channels is enabling major changes in student and teacher roles.
ing, you have to teach the student to learn without this computer also. And we're trying to get challenge them, they have to go out and study things, they have to find the answer out there, talking with each other or they are to do some kind of roleplay or they have to try to make a short film, or they have to do something like that. So they have to be the creator, not only this, yeah. And I'm skeptical if you just had to click this, it's like kind of activity but it's not a real activity, yeah. So we are doing both and I think you had a talk with Leonhard, and he is very, yeah, we have this pedagogical platform for NDLA, saying you have to activate the students and with that, I mean, we are not just teaching them to click but we are learning them to go out and find out and see the world and reflect, that's also important way of learning. Interview 13, 99-99

Smartboards
The use of smartboards is also linked to the question of student activity. For example, digitisation aimed at smartboards carries the risk of neglecting student activity.
When Smartboards did rise some years ago we thought oh that's a nice thing to just use for a presentation or something and so we think that smartboards are okay for starting your learning session or something but you have to leave the smartboard very soon and let the pupils work together and do their own learning and they learn through working and through cooperation and through discussing and through doing some things. A reference to the world of the pupils and its functioning is established through the cooperation with the statistics office (cf. Chapter 8). In sociology, for example, it is possible to deal with the environment on the basis of statistics and then compare these with their direct environment.

Localisation
In terms of relevance to everyday life, the localisation of learning content is particularly attractive. This is due in particular to the availability of space. In comparison, classical schoolbooks have to be much more limited. The lifeworld relation of the materials is of particular relevance to OER.
Digital media and OER do not automatically turn bad lessons into good ones.
Cooperation with local museums or universities has already been mentioned (p. 15).
A possible next step is to link GPS data with learning content. This can be online via map services or offline via excursions to historical sites or, for example, to locations where water quality measurements are made and then sharing the results online. Localised scavenger hunts, which are linked to interaction with the environment, are also available for other subjects (e.g. training in the hotel industry).
14.13 Students' freedom to shape their own learning process One of the pedagogical considerations mentioned is the freedom of the pupils to shape their own learning process and to take responsibility for it. This also includes the freedom to switch off the laptop and learn things in the real world. This, however, must remain connected to the materials. Likewise, space must be created to articulate needs and exert influence to prevent pupils from becoming material processing machines.
14.14 Teaching as a creative exchange process The statements on teaching, which is understood as a creative process within a group, also go in a similar direction. This process is to be organized by the teacher. Teachers must be supported by the school management with regard to the creation and use of freedom of scope or by the conception of tasks or the provision of digital tools.
The tools include Talkwall, for example, a software tool that enables interaction within the learning group (Audience-Response-System). Arguments can be gathered and exchanged and thus a deeper discussion can be achieved. This can be used, for example, in the analysis of news or in the interpretation of poems. The development of such tools is one of the tasks of NDLA's development department.

Range of offers
The question of the range of NDLA offers is linked to many other aspects. These include sustainability and maintenance as well as the involvement of teachers in the production of materials (paid and unpaid). What is attractive from a pedagogical point of view, however, is that the availability of space makes it possible, for example, to present different perspectives on an object. How is Alexander the Great perceived from the perspective of the Iranians? What are the different proce-Learning with OER is more than drag & drop and quizzes. Dealing with the subject matter and classmates needs space and structure.
A collective pedagogical platform enables an organization like NDLA to work on the basis of common values and to professionalize.
dures for solving mathematical questions? These can be presented side by side in digital space and enable different approaches or a more in-depth discussion.
14.16 Pedagogical platform as a basis for joint action Over the past ten years, NDLA has developed a central pedagogical platform that brings the core elements and beliefs together that need to be taken into account when creating materials. It is supplemented by additional documents dealing with individual subtopics, e.g. project teaching and problem-centred learning. The ideas contained in the platform are the yardstick for the development of new projects and ideas.
14.17 Transparency of learning progress A further point is the transparency of the learning progress of pupils, which makes a reflection of the own learning processes possible. This is another challenge to be considered when creating OER.
The range covered by large-scale OER platforms is closely linked to the issue of keeping it up-to-date.
A collective pedagogical platform enables an organization like NDLA to work on the basis of common values and to professionalize.

Navigating the material
In particular with large-scale offers such as NDLA, the question of how to navigate the material arises.
Many users access NDLA via Google. At first glance, this seems surprising and it has unintended consequences. Thus, there are contents which are integrated in several subjects. In 5-10% of the search queries you will find a resource in an unintended subject context. For example, when searching for a mathematical content, it appears in its application in chemistry. Therefore, it is an important question to which context the material belongs or can belong and which context the user is interested in at any given moment.
Our biggest challenge right now with NDLA.no is that we actually don't have a front page. There is a front page but according to usability there is none. Our front page is google.no and the way we are consumed is that the pupil, most of the users are pupils (like 80-90 %), they go to Google, they use some words to find some content and NDLA pages come up in that list of resources so they look into them and they click on one of them. And then they follow that link into one of our pages and if it's the page they were looking for they might complete the consumption of that page and afterwards they leave. And then they go back to Google or YouTube and then they do another search and then they might just hit on another NDLA page and then they go there. That is the usage pattern of NDLA today. And by using Google as our frontpage we have more than 60 thousand user sessions a day. What if they could consume all this material, also in the learning context? It should be consumed or put into the learning situation. Interview 3, 13-13 This shows that the design of contextualized homepages is a central challenge for interdisciplinary platforms with the scope of NDLA. Here, for example, short-term cookies could be used to store the desired subject context and long-term cookies to store the age/grade of a student on the user's computer. Contextualisation can also be given to the user's responsibility. For example, if pupils access the site via Google, they could be offered the different contexts of the resource and asked to choose the one that is relevant to them.
In order to improve learning in contexts, it may be helpful to display the content reached via Google, but also to refer to any necessary basics and further content. This can be done by editors, by computer-based recommendations ("Other users who have read this article are also interested") or a combination of both.
In connection with the design of the new platform, the extent to which subject-specific differences in navigation are necessary was discussed. Differences in the curricula structure between subjects can be of great significance so that curriculum-based navigation seems impossible. At the same time, it is necessary to weigh the individual preferences of the editorial teams against a lean software architecture that, nevertheless, meets the core needs.
We have some basic principles for the new design and that is... to create text or topical texts as an entry point for all of the topics, subject consists of. And then to have just two or three sort of content types connected to it, so that all the sites look the same and the terminology is sort of universal for the whole site. So that you create…., if you have learned to use one subject, you can use another. You can easily get used to using another and… we're in a process here, that we haven't sort of landed it completely. Because in a phase…., when you leave the old system, it's very easy to see: "Okay, we had too much of this, too much of this, we need to do this here. And in that sort of early phase, everything is very clear. We just need to do that, we just need to tell one story, yeah, let's do that." And then, when you start to plan for the next generation, you have to ask yourself: "Okay, so the simple story is…, that's the simple story…, okay, let's go over the simple story…. And then you build and build and …, but what about…? And then it becomes messy again. And the solution you thought it was there, dis solves in a way, because you have to…, well, that doesn't fit that subject and that doesn't fit that subject…" Users of NDLA.no often use Google instead of website navigation.
So, if you, if you, my sort of…, my honest opinion is that, perhaps we should not make all the subjects the same. Perhaps math, perhaps science should look different from culture and communication.
So perhaps the way this subject is built and the way it looks should reflect this sort of artistic subject, a scientific subject, or is it… So, it doesn't necessarily have to be the same. As long as it sort of projects some of the culture or the essence of the subject and as long as it's intuitive. That's one aspect of it.
Another aspect of it is, that the curriculum plans are very different from subject to subject. So in some vocational subjects, it is very down to detail. You must be able to weld, or you must be able to explain how this and this works. In those subjects, the sort of the curriculum entry system is very good. Another challenge is the presentation of spiral curricula.
Recurring topics are to be presented according to annual student cohorts and at different levels and linked with topics from other subjects. Especially when accessed via Google, the question is: how can I offer users the right class level/knowledge level and also show them the ways to foundation knowledge and advanced knowledge.
The different levels of difficulty within NDLA must be visualized.
NDLA offers many access points to navigate the existing material. Since the different users also use different ways to navigate the material, it makes sense to keep them. In the new version, however, one form of navigation is to be placed in the centre, while others are first hidden or can be found at the bottom of the page.
In NDLA generation I, we had lots of ways to find, you know, resources. You could go through menus, you can use search, you can use some of a curriculum guide, then you can use a way of content-type tree. So, they said, this not good, you have to decide on one way of navigation and stick to it. So, that's sort of what we're going to try to do in generation II of NDLA. Improve usability by cleaning up navigation, menus, and… sort of logic, because everyone agrees that it's difficult to find your way around in NDLA today.
I: But isn't someone relying on "search" and someone is relying on the "topic wise" search, and someone is relying on the other way of finding resources. And, if you just concentrate on one, won't you create some deeply unhappy users? The use of individualizable learning paths is one approach to achieving a clearer structure and navigation in context. This can counteract the fragmentation of learning content.
I think, for myself at the moment, the best way we have done this last year is this learning path. I think this is in the area we are working it's a better way of doing it and the users are supporting that because it's very fragmented, they don't see the connection between things. We have relations between that side but they didn't find them but now they see it, there is a connection and I think they are happy with that. Interview 13, 86-86 These can be flexibly adapted by the teacher to the needs of the students. In the sense of a "playlist", teachers can compile content and share it with students or other teachers. Since the content comes from NDLA and is properly licenced, it is easier to share. The material can be annotated with personal notes and notes for students. The learning paths provide transparency with regard to the following content, tasks and goals.
In addition to transparency for the user, it is also of great importance to provide the content with its context in a machine-readable form. For this purpose, the new software provides interfaces (APIs) that can pass the content on to other software systems. This enables other projects and countries to reuse NDLA content and to use their own financial resources for other tasks and contents. The contents can also be imported into learning management systems (LMS) such as "it's learning" or "canvas" via the APIs with their contexts. It is interesting to note that NDLA explicitly welcomes this further use of content. If the transfer of NDLA's content to other platforms leads to the success of the platform in question and to learning success for pupils in other places, it is a success for NDLA as well.
The material is structured based on the competence expectations formulated in outline plans intended for teach ers. A central concern for the further development of NDLA, however, is the introduction of competence goals that are comprehensible to pupils, too. NDLA has (for the time being) decided against the functionality of a learning management system (LMS). The existing login function was abandoned because the students could communicate with each other via this system. In order to be able to use NDLA for exams at the end of the school year, the function was permanently deactivated. At the same time, this means that no user data is stored. On the one hand, this avoids numerous problems with the storage of user data.
On the other hand, the question arises, for example, how teachers can then gain insight into the pupils' results. At schools in Norway, OneDrive is sometimes used as a storage solution to storing the results and make them available to the teacher. Alternatively, schools can use their own learning management systems such as "it's learning" and import content from NDLA (see chapter Interfaces).
A further problem is the intermediate storage of semifinished tasks. This is not possible because of the missing login. A URL would be useful in order to enter a successfully solved task again at this stage.
The discussion about learning management systems and the level of access a teacher receives in student data rais es the question of self-determined learning in contrast to full supervision.
[…] We have to give our students freedom, they are not, they can learn in another way also and we are going to give them that for but we are not going to catch them. Then I don't like this learning management system, like catching in everyday there is someone watching you. There is a big brother, who really is your teacher, watching everything you do. In addition, the question of how to deal with previous user data when a change of school or teaching staff occurs. Do they remain with the original school/teacher, are they deleted, or are they (partly) handed over to the new school/teacher or the parents? On the one hand, (positive or negative) attributions may be reproduced at the new school/the new teaching staff; on the other hand, without the data, it will not be possible to pick up from a student's previous learning development.
NDLA is intentionally no longer a learning management system and therefore usable for exams.
This user data is already available, but has so far not been saved. The h5p content passes on information about pauses, clicked content etc. to the embedding platform via an XAPI interface. But we just make this stream of statements available, but if nothing captures them, they disappear. So, we don't store them. And we just make them available. Some of our users are something called a learning record store. It's a standardize way to store these statements and analyze them, and definitely big data and you can have analytic engines and things coupled on to that. So, if we had this click to word support I don't think we have it in anything other than we have the content that could mark the words where you supposed to click words. But we could put it into articles so, if we make something for bigger text maybe it's a good idea to make it possible to click words and tell the system. And that will be up to others to do the big data and make use of all this data and use it to, offer the learner a better, or more suitable next steps in this learning path, or to improve the content. You can do a lot of things with it. Interview 7, 21-21 NDLA's new platform will be able to collect this data.
The more comprehensive such a platform becomes, the more problematic will be the storage of these data on the individual level. It would be possible to create comprehensive interest and competence profiles of learn ers. Since there can never be a completely secure protection of such data, this will always be a problem. In addition, numerous companies have an interest in the legal exploitation of such data. It is pointed out that the data stored on learning platforms should belong to the pupils. At the moment, students often give their data to platform providers who are mainly concerned with economic exploitation.
The right and the technical possibilities to have access to one's own data and to be able to delete them will be of central importance.
The NDLA login features that are no longer available are considered nice-to-have, but not really necessary. This is also due to the difficult legal situation.

A:
That is a possible trap. If you develop too much log-in feature and store data and it's not traditionally secured in a way. That is a trap. But I'm not sure that we need all that log-in activity. I think that you can, if you just present tasks and resources, you can find some visions like, you know, send it via e-mail and store it on your computer or… you don't have to have all that interaction. It's nice to have but it's not strictly necessary. And if you don't have clearance or if you are not within in the law, of course that can be very serious. Interview 12, An informed consent by young people or their legal guardians seems only conditionally possible, due to the incalculable consequences of consent or refusal. It would be conceivable that future employers would demand access to this data as a "voluntary" part of application documents.
The presentation of adaptive learning materials is attractive for many of the interviewees, but it also entails risks that cannot be ignored. A negative assessment of the learning development of students carried out by software can lead to underestimating their ability and thus systematically to their being underchallenged.
Big Data usage of OER is possible without a login and useful for adaptive Learning.
We have to sort of end up…, we have to…, you think about the next generation, next generation. We have to sort of end up somewhere, where everything is very customizable. The sort of the student or the teacher that say exactly what they want, what level they're on, what they find difficult or not, and then you get some kind of customized experience on the web.
But there's also some sort of Pandora's box here. If you have a 10-12 year old student in 10-15 years time…. At some point of time it sort of takes a test. And they find out, you're not very textual, you're a more visual type… you don't like this, you don't like this, you don't like this.., so now we're going to streamline something for you.
There's sort of horrible thought when it comes to individual freedom and the fact that we change and we… I mean, if you don't like reading texts… you're a visual guy, does that mean, that we should feed that feature in you all the time? Or shouldn't you be exposed to a lot of text? So it is, when you talk about customization… it's a very fine line between sort of predicting, that you're not going to be a new Einstein, so we're going to feed you with simple tasks. And you're 12 years-old and in two years you have developed extremely, and perhaps, it will become Einstein… So, that's the really scary part about customizing, about sort of…you said, 'it'll be too much, yeah we'll customize it for you'. But how to make sure, that that process will enrich your life and not limit it? That's a huge question. Interview 12, 87-89 A meaningful use of the resulting data can, however, also take place without storage on an individual level.
I: Well, for example if you look at the short stories and you have this flip through… and I thought it would be nice to have a function that you can hoover over a word and you get the translation and on the same time you have a big data algorithm, that says 'well, this word was hoovered over by 10.000 students'. Are you planning something like this for the upcoming version or is it…?
A: Yeah, we are working on that. That's not specific to the upcoming version. We're already mapping a lot of sort of that types of activities. Like we can see all the incidents of copy and paste, or not to paste but the copy. Whenever something is copied we can use a Google Tag Manager and that will sort of be one event. So, we don't have the hoover facility but very many other things we can map, like where they click, how long they stay on the site, how far they go when they drag a task before they quit or how many complete the task. Interview 12, 137-138 The data collected so far can be analysed across all persons. Thus, problem areas as well as different learning types or location-or time-dependent usage patterns can be identified.
The question of who has access to this anonymous data remains unanswered. For the development of new content, it might be useful to give cooperating companies access to it, since they can then recognize which content is accepted, which words represent difficulties or which task formats work for which groups of students.
So, I am identifying points of which books actually work better. What you can do is of course you can identify, if you mine data you can probably use say ten thousand students going through my books. I could identify which tasks that work, where the difficult words were, horrible explanations, bad pictures: I can again take them away so I get the essence of a perfect, fantastic school book.
And of course those data would be fantastic for me. And say, for example, the data says that your 7 year old boy has problems reading so he can be identified and helped, that's also, it's fantastic.
Responsible handling of user data in OER as well as the avoidance of full monitoring of usage is important (Big Brother).
And that sort of says 'okay, go ahead and share this data' because it would be fantastic for all users, all producers of teaching materials to make the perfect teaching materials. That's specially sewn for every individual. Bad part is of course that you stack up huge amounts of pupil data. And those pupil data if it's any open domain could be used to anything. So, insurance companies and stuff like that. I think the real big problem is in the in the municipalities it's of course in everyone's interest that both data and information is free. Technology also is free. And municipalities love open source for example. And I think that's an excellent idea but what we we're struggling to find out is how to secure the data but also share them to sort of correct… if we could figure that out that would be fantastic. Interview 10, 51-53 On the practical level, this adaptive differentiation is made possible by adapting to the results of the students, for example by introducing new content types through h5p.

User-oriented design
One aspect that has been mentioned several times by different people is the user-oriented design. In this respect, the question arises: Who are the users? Until 2015, the focus was mainly on teachers. The development of the students was discussed with them. From 2015 onwards, more attention was paid to the pupils and they were involved in the development of the materials through interviews, observations and tests.
That's when we sort of changed focus because until 2015 we always spoke with the teachers and to teachers about students but not directly to the students and then suddenly we thought, hey who is using this? It's actually the students, maybe we should ask them what they want, how they learn. That's the new approach and last year we had 34 classes of students, actually it's as many as 1700 Students, we had some internet-schools as well and we used them interviewed them, observed how the teaching was done and cooperated with them to find out what is good with NDLA and what needs to be changed. What can we improve? We have done that this year as well and not just talking to them and observing but actually done user-testing like tasks that we asked them to do on the computer and then observed and film and used eye tracker to make sure, you know, because both students and teachers are normally very nice. They say well, we have a lot of nice things, it´s good, but the students they do things we see well it´s not that easy, to find your way around and we have too much stuff and all these things. That's a very good thing for us now that we are actually comparing, work ing on, I guess the others have told you that as well, our new design that's supposed to be ready 2018. Interview 11, 5-5 This change of perspective is not easy for an organisation or its individual members.
I: You said, you switched the user focus from teacher to student. So do you do both now or do you concentrate on the students?
A: In the beginning, we thought that we had to just really make us aware of the students, we had to do mostly that but of course we can't forget the teacher, so this year we have sort of taken the teachers approach back or just we do both. But just for us to make that shift because most of us who work at NDLA, we are teachers, we worked a long time with students. I think we had, that includes myself, had an idea that we knew what the students wanted, we knew how they learned, we knew so many things, but obviously we didn´t know everything. The developers, who thought they knew the right way because of their many years of experience, began to question their expertise after having had the chance to test materials together with students..

A:
We have people who has been working on traditional learning resource development, modelers, for the past 20 years. And they are kind of setting that way of thinking. So the teachers know the pupils, they know The usability of a OER platform is central to students, teachers and editors.
better than the pupils how things should be presented and what they should learn. They know their experience, they know so much about pupils so they can just tell them what to do, and then they will learn and so on.
We have to challenge that kind of culture. Best way we experience to do it is that 'okay, let them makes some resources' and then we put them into a user-interface and then we send them out to the pupils together with professional pupil doing user-testing. Interview 3, 9-9 At the same time, it is necessary to keep in mind the needs of both groups: those of the students as endusers and those of the teachers as gatekeepers -as well as those who are to make adjustments to the materials in the interests of their students.
In order to keep an eye on the manifold needs of pupils and teachers in the course of development, different fictitious personas were developed, each with detailed descriptions. Every persona is encouraged to question how materials must be designed to meet the needs of the respective user. The personas are publicly available at https://brukskvalitetsplattform.ndla.no/personas.html (translated: https://is.gd/ponomi).
But to individualize the student we have made something called personas, I don´t know if you know what that is?

I: Yeah, I worked with design thinking in Germany
A: Well good. We actually created sort of 'real' students for us to have them in our mind all the time. So we have to think about this and we have to think about that. So we are two teachers, so how would Monika react to this resource or how would Helge react because he is a vocational teacher, where she is a academical and she loves our computers and IT and he just hates it, for the students as well. She teaches her students on health and social and she struggles with so and so. He is a student at, lets say, math, higher math and physics. How would he react? To force us to actually think about real students and real people and have that with us all the time when we produce stuff. Because it´s so easy to think of them just as a group and there is such a variety and we need to focus on all kind of things that are important. It´s not just that we have students who are functioning well but we have students with all kinds of problems, it might be visual, hearing. To think about these things as well, concentration issues, just anything. Of course we can´t just create 200.000 personas, with all the students that we have in Norway, but some, some typical, some bigger groups. I think that has helped us in our work as well. Interview 11,[25][26][27] This is a way to take into account the diversity of students in terms of gender, cultural background and academic achievement. A: At least we focus on gender for one and then we focus on vocational versus academicals so we have three of each. I am working on some new now, because we didn´t have them all. We have the difference between the students who are really the topnotch, who want to become lawyers and some of them struggle with how much should I read, how can I really achieve those top grades. The really weak, the students who struggle, because they haven´t in middle school or in even primary school, they haven´t gotten the basics of math or Norwegian. They sort of take their problems with them through the system, so that´s a difference. Also Norwegian / Non-Norwegian, like saying you are from Syria or another country, who have actually traveled and struggle with, I mean some of them struggle with the language and everything. Others can be really smart in math or in English and that kind of stuff. Some of the Norwegian subjects, such as trying to learn religion or any other subject in Norwegian is difficult because of all the words that they don´t know. That's another dimension than being practical and more theoretical. Also we need to have some underachievers, because there are some of them as well who only go to school, like a lot of students: How little can I do to get good grades, sort of. I think those are some of the dimensions anyway. And of course age. That´s what we call version one. We need to work on more as well, to add more to it. We started there and that was ... yeah. The work with personas, test classes and the focus on the needs of the users draws on the methods of design thinking, which, as described in Müller & Duong (2015), also have their appeal for pedagogical scenarios.

I:
User-oriented design is a focus of the work being done on the new platform. This includes both end users and contributors. One pointer for the usability question was to follow the processes of Google and Facebook, who have invested numerous resources in developing the most efficient interfaces possible.
With regard to accessibility, the Norwegian government has stipulated that from 2018 onwards all new materials should be accessible to pupils with disabilities. From 2021, this will also apply to existing materials. These are currently being checked with the help of the WCAG standard. In order to provide subtitles to all existing videos by 2021, each day four videos are subtitled in the two Norwegian languages.
h5p also endeavours to make all content types barrierfree.
Using personas (fictuous pupils or teachers described in detail ) helps the editors not to lose sight of and to understand the diversity of the students in the creation of OER.

The technical framework conditions
With regard to the technical framework conditions, the interviewees addressed the new platform, on the one hand, and the simple usability by content creators on the other.
Concerning the metadata, a new standard (NS4180) will be introduced, based on the preparatory work of the Learning Resource Metadata Initiative (LRMI) of the Dublin Core Metadata Initiative (DCMI). The core task of metadata maintenance is to communicate the benefits to those who create content. This may sound trivial as any teacher should understand that retrieval is crucial. However, too complex metadata forms can discour age contributors. For example, Cerpus -a cooperation partner of NDLA -uses Dewey to classify the content of courses. Moreover, the question arises whether one should reinvent the wheel or rather build on existing systems. The use of existing international classifications makes it possible to find the contents in other languages and thus, if necessary, only have to pay for the translation.
Preset metadata (e.g. based on previous posts or the environment of creation) can facilitate the process of creation. For example, a math teacher is very likely to create other materials for mathematics, and materials within a learning path are usually dedicated to a common topic.
In order to make qualitative statements about the difficulty of content, Cerpus uses the revised concept of Bloom's Taxonomy. These data are collected in dialogical form in order not to overwhelm the contributors. The Belgian OER platform "KlasCement" was cited as an example of an understandable, dialogical user guidance in respect of image licences.
I: So, well especially when you come to images or stuff like that, you have to know about the license. A: Yeah, but "KlasCement", look at their solution, because you get to upload a picture and then when you click "save", you will get a human question, not a technical question. But sort of a question that a human 's going to understand. Then they ask you: "Is it your picture, yes or no?" A role-based, simplified and dialogue-based input option allows less tech-savvy teachers to collaborate on OER without scaring them off.
"No? Okay, then where did you find it?" -It's Google's picture. "Okay, does google own the picture, or is there?…".
You know, it's great. And of course it comes right after you load the picture, not before. Because, when you get all those questions before you load the picture… you're not going to waste time… But you have loaded the picture and you're that close… and then you have three or four questions and the last one is great… it's something like: Not sure? Then contact us.
So then, 78 percent of all the uploads can be fixed through a couple of questions and those who can't find out after those questions, they can contact the help desk. So, I think that's a good system, sort of questions. You do something, and you get these questions and they're not in technical terminology. Interview 12, 80-80 The design of these processes has a direct influence on further cost structures. Particularly in larger facilities, unnecessary, mandatory operating elements (e.g. checkboxes, clicks) result in frustration and lower productivity.

A:
One of the mistakes we made was perhaps not putting enough effort into creating a publishing system that was sort of efficient enough. Lots of forms and fields, it was too technical in a way and you had to click and click and click. And when you have 50 people working, if those 50 people work in a publishing system that is outdated and not sort of customized enough, you have to make a lot of unnecessary clicks, if you add up all time spend on clicking unnecessarily, you would probably find that 'okay, you have spent millions on working hours' and frustration and, yes. But you don't really notice that because those fifty people they are sort of sitting at their own offices in separate parts of the country complaining at different times. So, that is kind of sort of a leak, it leaks a little bit here and a little bit there. If it had been a big leak somewhere it was like 'oh, we need to fix it, watch!' But because this is so spread out at a sort of a low level in the organization, it is not attended to sufficiently. So, that's one of the things that we could learn from, is to sort of create an efficient production system so that those who are basically teachers, pedagogues, could focus on what they're good at. You know, producing subject resources and not clicking in dropdown menus and entering Google Tags and… you know, other people can do that, but let them write or create resources in an efficient way.
I: So, you would split the task for example and say, 'well, someone creates the resource and somebody else takes care of finding the right tags or whatsoever'?
A: Let's say we have an English teacher working for us. We have hired him because he is creative, because he knows the students, he knows his topics, he knows how to reach through to students, he writes well -why should we try to sort of teach him all the other aspects of publishing? We should use him for what he is good at and sort of clear away all the other noise… let him be the good writer he is.
I: So, would you say it would be better to let him just write and 'here is picture of Big Ben' and somebody else searches for a free picture of Big Ben or provides him with three pictures and he can choose or anything like this?
A: Yes, I think so. And I think we should have a specialist on Google, or our search engine optimization, so that when we produce and publish something, it should go through also a specialist that checks that this resource is tagged for optimal searchability. I don't believe that you should, those that you… hire into editorial staffs, I don't think you should make them become specialists in all this other areas as well. I don't think that is an efficient way of working. Interview 12, 153-158 Such a distribution of roles links aspects of professionalisation with the technical design of processes.
Preset OER metadata in the creation of OER can be helpful in getting teachers to take on this task rather than ignoring it.
The software used determines the technical framework conditions. The decision for the content management system "Drupal" was at the time correct in view of its quick implementation; nevertheless, it lead to numerous problems.
This included having to take into account the individual wishes of the various editorial teams, which led to an increasingly comprehensive expansion via modules that made the system slow and maintenance-intensive. This led to numerous crashes and data loss.
For reasons of clarity and to avoid similar problems in other countries, the decision-making process is described in detail in the following. At the beginning of the project, the ministry representatives were concerned that NDLA was only setting up another platform for sharing content. Therefore, they referred NDLA to a company to build the platform for the project. The editors of the first three subjects expressed all their wishes.
Based on their previous experience, the company opted for Drupal 5 as the underlying platform and an existing community. However, Drupal 5 was only suitable for a limited part of the requirements. For the other requirements, numerous modules were added, each of which fulfilled only a limited part of the requirements. This made the system unstable and difficult to administer. At this point, NDLA was given the responsibility again and started looking for a partner to further develop and stabilize the site. Subsequently, the system was updated using Drupal 6, which remained the basis of NDLA for a long time. The update process was extremely complicated due to the numerous integrated modules and editors' different design requirements. The subsequent maintenance of the source code was likewise complex.
After a period of ten years, however, the needs of NDLA (both end-users and editors) can be described much more precisely than would have been possible previously. NDLA's new platform therefore takes up these aspects of process design and many other aspects from NDLA's ten years of expertise. The platform is developed as an open source, is designed for multilingualism and can therefore be used by other countries free of charge. It can be downloaded from github (https://github.com/NDLANO). Such a specific platform is more expensive than the use of a CMS, but this is exactly what makes the Norwegian software so useful to build on instead of developing one's own new solutions.
Another expectation attached to the new platform is that it is likely to improve the possibilities for external cooperation. This includes on the one hand the structured collection of feedback and suggestions for revision and on the other hand the quality assurance of external contributions by the support team.
The technical requirements have also changed significantly over the last ten years. While ten years ago it was common to minimize image sizes in order to save bandwidth, a challenge today is to adapt resolutions to the device (up to 4K displays) and the available bandwidth.
The newly built platform of NDLA.no is available as open source on github and is already geared towards multilingualism.

Quality assurance
The question of how quality assurance can be designed in Open Educational Resources is a focus of the NDLA colleagues. The question not only includes the aspect of updating content and correcting errors, but also the verification of external contributions.
The quality of NDLA was assessed very positively in a study encompassing 1,600 teachers in Norway.
The topic of quality has already been addressed in connection with the market (p.10). It should be emphasized that NDLA is willing to pay for high-quality content. However, this requires a detailed description of what is to be understood as high quality beyond the required openness.
What is interesting here is NDLA's understanding of quality and the resulting objectives: So, we don't have goals that everything should be as good is it could be but it should be as good as it should be. Interview 1, 23-23 This perfectionism limiting, overarching requirement is substantiated by five dimensions of quality, which were seen to be essential in the course of NDLA's development process.
And then we have these five quality dimensions. One of them is subject quality, one is pedagogical quality, one is technical quality, one is the user perspective quality and one is production quality. So all of these five dimensions have different representatives in the organization. And we have guidelines on how to achieve continued improvement on all of them. Interview 1, 141-141 As mentioned in the interview, the five areas are represented by teams, each of which has its own leadership and is also represented at weekly meetings.
In a large-scale organisation like NDLA, there is now an internal support structure that assists editors in creating offers of high quality.
But the year later we rearranged our organization and I became a team leader of the support team. You have seen our organization; we have, like, a circle where we build and then we test and then we change and go along like this and we have also a process to try new stuff inside, and that is explore, testing and if they are good enough then we build on. It could be technical stuff to build or it could be a lesson. But my team is the middle, we call ourselves 'RoS' (Counceling and support) and we support all other teams, so we are 12-13 people and I am the team leader. And then we have some specialist like a photographer, he travels along and takes pictures, of the right licenses. But we also have one, do you know scanpix stock photos, where we buy pictures, and there we have one person to find pictures, so yeah. It's called Scanpix. And then we have all people in Alsvag, we call them desk (like desk publishing), before we send something on air, we take a quality test to check if the license is right and the image is big enough, the colors are nice, you know, lots of technical things we just check off the list. And every Thursday it goes on air. So, then we call them deskthat's the team with a leader, so they are 4-5 people.
There is also support of production, like if we have an editor, who would like to make a quiz, she could just set up questions and answers and then some other had to put it inside. Interview 4, 6-6 The question of a common style is addressed by this "Desk" team and the guidelines from the pedagogical platform.
New team members can find information pages on the "brukskvalitets platform" (https://brukskvalitetsplattform.ndla.no/) that provide an introduction to the following topics, which are directly linked to the quality of the materials: On these pages there are numerous further notes on the design of OER materials, which were not all discussed in the interviews. New projects can profit from these references (even if they do not speak Norwegian: https://is.gd/etenad).
Errors can not always be avoided, even with the most comprehensive quality assurance measures. Information on the correction of errors or on update requirements are already provided by the users. Depending on the scope of the problem, these can be corrected by the person processing the mails directly or passed on to the leader of the support team. The management of support requests is now handled by zendesk.
The analysis of the feedback is not limited to the processing of the direct requests via the contact form. Instead, social media are used to an enormous extent for quality assurance and exchange with end users. This includes the fact, for example, that each subject has its own social media channels.
But we also, in a large extend, we are very hard listeners, we want to learn from the usage from the students so we use social media a lot and we are very active on social media so every subject has its own channel on social media. We are also very active on YouTube, we try to reach the students on our YouTube Channel, which is very popular. Somebody makes the video on NDLA we also make it available on YouTube as well. Sometimes because of the right we cannot do it, but most of the time. And we also use a lot of statistics to try to understand how our students use our pages and try to improve them based on our traffic numbers, it's not actually learning analytics but it's still something we are trying to have the focus of the user not just on the user and try to find out how the world looks from their perspective. Interview 1, 23-23 A third source of feedback are the cooperating schools and their 1,200 pupils.
We have this pilot school arrangements, last year we had 1200 students enrolled, to give us feedback. And that is structural feedback, we actually visit the schools and work with them. Interview 1, 23-23 On the one hand, this enables feedback on existing offers, but also a user-oriented, iterative development of new offers (see p. 50).
There is also a discussion about quality for shared content.
Here, too, the question of perfectionism must be clarified.
That's the way we create quality resources, by sharing it. And by quality I mean, I'm thinking more in terms of "is this relevant to the students? Is it fresh? Is it interesting?" You know. So, if that teacher goes down to the workshop, makes his video, it's fresh, it's relevant, it's close, it's made now.
Interview 12, 25-25 The teachers themselves also think about wether their performance is good enough: Teachers are afraid that what they produce is not good enough. If it could just be an idea or one task or nothing that lasts for 6 weeks just a small little thing. I think it would be easier for teachers. But I have this great idea, it would take the students 30 minutes to work on but it´s still good. So that the difference between what's good and what's not good enough and the quality aspect is also a challenge that we need to think about. We also need to respect teachers as professionals.
They have an education. We need to trust them, that what they do is actually good enough. Interview 11, 37-37 A special proximity to the subject matter or a special up-to-dateness may be more important than a perfect production quality. An example from the Germanspeak ing area of the importance of the relationship between the teacher and the students, the charisma of the person and their enthusiasm for the subject is the You-tube channel Grammar rocks!!! (https://www.youtube.com/channel/UCgB2mCGCzUsgJe6cJjqyMXA).
Nevertheless, the question of quality assurance with regard to shared content remains a challenge. One topic of the interviews was the creation of a "Circle of Trust" function.
Colleague A trusts colleague C and editor D. They both trust in the work of Colleague B. In a "Circle of Trust" model, therefore, colleague A would also report colleague B's materials as trustworthy.
This has already been tried out at the personal level in sub-projects.
Since all teachers in Norway have a personal identification within a nationwide login system, such a solution could be implemented relatively easily in Norway.
When it comes to quality assurance, it is particularly important to raise awareness among teachers about licensing issues in order to avoid copyright infringements. A peer review would also be an alternative to a review by editors, who would be overwhelmed by a larger amount of shared content.
Another interesting question is how pupils can be involved in further development of materials. NDLA still Norwegian teachers are afraid that their work might not be perfect. However, one has to ask to what degree perfection (beyond correctness) of OER is necessary for the students.
has some reluctance in this respect, because they are under special observation and cannot jeopardize the success of the project by being too open. It would be conceivable, however, for the pupils to report back by means of a +1 button in the learning path, where they should mark the part of the learning path that has helped them the most or click on the material where they see the greatest need for improvement.
Another question in the field of quality assurance is: How can materials be kept up-to-date over longer periods of time? NDLA is addressing this challenge at various levels. The creation process of many materials goes back to three-year-old editorial teams. These editorial teams are dissolved after their work phase, their members turn to new tasks or return to their original schools. As a result, the materials produced are exposed solely to the wind and weather of the digital world and run the risk of losing their reference to the analogue world. This danger exists to a similar extent for schoolbooks.
NDLA has taken various measures to better address the problem of updating in future. These include a reminder function so that materials are given a date when they are created at which they are to be checked. In addition, the integration of content from external cooperation partners (e.g. the Office of Statistics) ensures that materials remain up-to-date. External partners provide data via their own interfaces, which are then embedded in the current versions of the NDLA materials.
Another approach is to separate texts into more consistent sections and more easily updated parts. These are, for example, introductory paragraphs with current references.
Interesting are the remarks of a cooperation partner regarding the effectiveness of the use of funds concerning the quantity of high-quality materials that were created by the financial support: In my opinion, the point is: "Bang for the buck", to get most out of your money. So if you're the government, what would you going to do? Pour money into public companies or sort of pinpoint the money into a company that doesn't need to make new money or make revenues for stockholders. I think even though NDLA has probably cost in less than ten years 700 million crowns, if you put those 700 million into the open market and said 'Go ahead we'll buy this and that'. I think that the bang for buck that you'll get would be much, much lower. Because here you have an organization that doesn't need to think about revenue. They can just focus on one thing only and that is the quality of content. They don't have to think about sort of going bankrupt or losing their job. So I think it will be almost impossible to get 700 million kroners worth of editorial material or textbook material that would cover all 42 subjects. I mean NDLA covers stuff from smithing and all kinds of stuff that nobody even thought about making teaching materials for.
Interview 10, 32-32 NDLA's new technical platform will include reminders to check whether topics need an update (a best before date).

Professionalisation processes
NDLA is of particular interest to other countries because of its long duration, the wide range of offers for upper secondary education and the experience gained. This explicitly refers to the professionalisation processes that the organisation has gone through. The continuous further development of the organisation appears to employees as both an opportunity and a challenge.
NDLA positions itself in the existing system as a virtual organisation, which acts on behalf of the political leaders and the representatives of the 18 counties (see page 8). It is in close contact with the schools and acquires content via the market. Teachers from schools are particularly involved in these processes.
Internally, a distinction is made between two levels at which teachers are involved. Firstly, the level of systematic material production and dissemination and, secondly, the level of voluntary revision and sharing of own materials under free licence. While NDLA is very successful at the level of systemic production, the second level is still expandable.
The systematic creation is a clearly structured process. Based on the assessment of needs by the NDLA team (e.g. through feedback from schools or analysis), the Board of Directors decides on the choice of subjects for which materials should be created. After this decision, teachers are sought who apply to belong to the respective editorial team. Teachers are seconded for a period of three years (30% of the budget). The editorial team is given a leadership and is provided with a budget (e.g. 1 million € for three years). The requirements for the materials are defined on the basis of the curriculum. Public institutions such as universities or museums are contacted whether they would like to contribute content.
The editorial team then considers which content it creates itself and which content is put out to tender to be created by the market.
70 % of our budgets is spend on procurement and you are going to meet two of the vendors. And the rest, 30 %, goes into paying for the teachers.
Interview 1, 20-20 Further information can be provided on the individual process stages.
The choice of subjects, for example, is based on criteria other than those used by a schoolbook publisher. For example, it is possible to pick up subjects for which there is little demand. The example of the Sami language illustrates this.
And we got one subject called 'the south a lap Language'. We got eight pupils in Norway but we got to do the materials for those. And it costs just as much for the eight people as it does for the subject that covers about 20.000. So what do I have to consider? The board would like that we go into the subjects that cover 20.000, 10.000 and so on. Also got a lot of pupils. But aswell we have to think about those who does not that, the market wouldn't supply them with learning materials, so we have to consider that as well. So, how to find a balance between the small subjects or the subjects for fewer pupils and those with many pupils? And as a background we got a lot of wishes coming in from the teachers: 'Oh please can't we have materials in my subject?' and so on. So, it's just a mix, so considerations, balancing, looking into how many pupils there are, what have the market supply the subject and so on. If you're going into the material that we've got now you will see that we've got materials in the big subjects and some small subjects but I think that the big subjects are in majority. But anyway, I think in the future we have to create material for more and more of that the small subjects.
Interview 5, 77-77 The materials for the Sami language were paid for with additional funds from other sources. At the same time, the editorial team benefits from NDLA's experience, infra structure and reach.
The Board of Directors also takes into account the schools' wishes for new materials and revisions.
This balance between subjects that attract larger and smaller student numbers is also interesting for Germany, where less popular subjects (e.g. computer science) or local languages (such as Platt or Sorbian) could benefit from such an offer. NDLA has started with Norwegian, Healthcare and Natural Sciences and now covers 80 out of 330 subjects at upper secondary level.
Interesting are the varying recommendations as to which subjects other countries should start with. One interviewee argues that one should start with marginal subjects, such as violin maker, and convince with quality, only supplementing the core subjects on demand.
To avoid sudden death, I think you should focus on subjects that are not a money machine for the publishing houses. Let's say you have, in Norway if we had focused only on some vocational subjects and some for specialized upper secondary, we wouldn't probably have had all the fuss we've had with the publishing houses. The Norwegian state is by law obliged to provide everyone with learning resources and in Norway for a lot of the subjects it's not commercially interesting for the publishing houses to create those resources. So, I would say to the state you have an obligation to provide the students with learning resources. The publishing houses won't do it because it's not commercially interesting. We can do it, we can organize creative commons, share, open, human rights. This is also about human rights, the right to free education. And then I would sort of build up the organization and then… teachers teaching other subjects would perhaps…[say:] 'You're so creative, I so much like what you did in that subject, why don't you do something like this in our subject as well?' And then you can sort of spread and gradually eat into the markets of the publishing houses. But I wouldn't go sort of head on with the publishing houses.
I: You wouldn't start with German, Math, and English?

A:
No. I would start with violin tuner or something. Because if you create good educational resources in a small subject, you will also be able to do it in a big subject when time comes. I mean teaching is teaching, you have different subjects but pedagogical part of it is… and the tools for presentation production, user experience, organization, I mean… so I would start.
I: That's quite interesting because usually people would say 'well, we would start with German, Math and English because it's the biggest market and it's the most interesting for schools to have materials in this field… .
A: I would start with the small subjects and I would do it damn good and I would sort of create this very enthusiastic group and then I think you would create the demand from a third party of from others, from students, from teachers. Why don't you to make something for this as well? We'll pay lots of money for this and you can produce for that subject for one tenth. So, come into it for us as well. But.. don't go in there [to] sort of to go after the publishing houses, go in there as a sort of… your agenda is to create great learning resources for students. And if you can sort of prove that you can that in some of the smaller subjects, then let other people draw the conclusion 'hey, why don't we do this in German?' And then The starting points in the creation of OER in Norway were the subjects with large student numbers.
they look at you and say 'okay, we invite you to..' instead of you barging inside 'hey, now we are going to do... ' Interview 12,[123][124][125][126][127] Another approach is to start with the core subjects and integrate smaller subjects later.
When setting up a new team, the team leader is hired first, who then selects the other team members together with the NDLA management.
The concept development process has developed further in recent years. While at the beginning it was only the task of the editorial team to develop a "script" for the subject, now the interested public is also included in a workshop format.
Earlier there has been a more traditional project or method which we call it 'Dreiebook' in Norwegian it's like a concept gets to start when several quality criteria of how should our new subject be build and a project manager etc. similar to commercial companies. Our success the 3-4 last years has been to invite all parties with interest in communicating knowledge within a subject to the society to workshops and to collaborate on building this. We did this on the history subject (history one and two), we started that with religion now and a sociology-similar subject. We will probably follow that origin that means network production, conceptually quite similar to what Alexander Osterwalder did when he built the business model generation book. He was the moderator of 450 editors. Interview 3, 49-49 Cooperation in the editorial teams enables a four-eyes principle in the development of materials at the subject level.
A critical dimension of professionalism relates to the handling of tenders. Due to the fact that 70% of the available funds are returned to the market, a legally sound tender for these funds is indispensable. This requires support of the editorial teams on the part of a competent contact person who mediates between pedagogical dreams and legally compliant tenders. NDLA has created standard processes for this, which point out the requirements for a legally compliant tender.
A further form of structured cooperation between teach ers is project-related involvement via work contracts. Here, teachers receive 3,000 € for the processing of a fixed work package and then have a period of one year to complete it.
And they have a lot to do every day but there's a program where we are taking these people into NDLA for some time and let them work with us. Then they are open and sharing in the creative commons license. No one say 'no' and they were happy to get allowed to be along with NDLA, I just interviewed 20 people last week and everyone was, they got some very small paying for what they're doing, but they also happy "Oh, this is what they dreamed of, this next year will be the good one" is what they're saying. So I think that way they want to share and then they know, there is some one looking after the spelling and checking something and they can ask for help and then they had the courage to publish. Interview 13, 54-54 Regarding voluntary remixing and sharing of content by teachers, it has already been mentioned that NDLA is not very successful in this area. Several ideas and initiatives have been developed to change this. This is particularly due to the fact that both the development and the maintenance of materials for all subjects (400 subjects in upper secondary level II including vocational training) cannot be realised using the previous model with editorial teams.
Focusing on the niche subjects (training for violin makers) in the creation of OER could initially help to achieve peaceful coexistence with publishers.
It´s not really a difficulty, but when you have 400 subjects you would want to create them all and quickly. But how we do it today, like we have people working on a subject for 2 years. It takes too long, so to figure out how to involve the crowd, anyone, all teachers, I mean there are tons of good teachers out there, who could actually work with us. And how we could create learning material in a new way. So that we would get easily a lot of subjects. Interview 11, 37-37 With regard to the causes for refusing to share selfcrea ted content, the fear that content could not be good enough is pointed out. In addition, content is locally being shared via LMS or Dropbox, as can be observed in schools in Germany at which copyright issues are largely ignored. This also applies to the remixing of NDLA content. This is (to a large extent) carried out without the consideration of licensing issues and is therefore not publicly shared. It would also be conceivable to reward active teachers who share content with flexible days off 2 or by taking such cooperation into account in the official assessment of their work.
One approach to attribution is to make the creation of content more transparent, so that it can be taken into account more strongly in recruitment/evaluation and so that contributors are proud of their involvement and better appreciated by employers.
The Belgian KlasCement platform allows teachers to download more materials when they share more themselves.
Another suggestion to motivate teachers to share content is to start with a limited number of high quality materials for a subject and then add more.
To cover the full range of subjects (400 subjects in upper secondary education, including VET) with OER, the previous approach via editorial teams does not work.
Here a crowd-based approach would be required.
I also think that it would be very interesting to look at a model where NDLA sort of creates some core resources. Have you ever been to….., in Scotland they have these narrow fjords or passages where seawater passes through with very strong currents. They have these chains, that they put out there with some corals on them. And then the nutritious seawater flows through and more and more corals build up.
That is kind of a way, that I think it can work, if you create in a subject let's say 20 resources, that have a high quality and that you know that the teachers will look for. So then you sort of put these out there and then, as students and teachers use these resources, you use google analytics to see: Here we have lots of users, here we don't have as many, what's particular about this one? And you also have contact forms on the websites and you ask around: "Did you use this one, or did you use this one". "Yes, I used that one, you should also make something on….". "Okay, would you like to contribute?". So you put something out there and you make sure it's being used and you have supplementary stuff resources that are attached to it, which may have poor sort of quality, as such, but, which will provide a choice for other users and diversity. I think, that the term quality is often a sort of stumbling block, when we talk about learning resources, because so many people feel that it has to be very professional, with two or three teachers… and the translator has to look at it. And… it has to be language vetted, it has to be so, so good. I'm not sure that it sort of pays off with the students in the classroom.
Interview 12, 21-22 Here, too, we find ourselves in the quality discussion again. At the same time, the free licence of the content allows an adjustment at any time if content still fails to conform to the standard.
Especially when it comes to teachers sharing something in their spare time, it is necessary to make it as easy as possible. In order to lower the barriers to sharing content, reference was also made to the design of input masks (see p.45f ), which must be geared to the needs, language and skills of the teachers and relieve them of the tasks that take too much of their time.
Another approach is micro-editing: if it is only a matter of adapting a question or a section of text, users should be able to quickly and easily make suggestions for changes, which are then checked by the team. Attempting to do that without a login is one way to make it as easy as possible.
Now it comes to the time aspect. We have to create some solutions. Today, you have to sort of log on to websites and fill in lots of forms. It has to be a lot simpler, in terms of the contributors being… they read something and they say, okay, that's a darn question, it's not good at all. Then, if a teacher reads one of our resources and finds they a bad question, you should be able to click directly on the website and write a new version of it and press share. That should then go to our desk or one of the editorial staffs, as a suggestion. So, I think you could have…, we could at least try out, to let everyone do it without even logging in. And it just goes as an anonymous suggestion. If that creates too much spam, we have to have some kind of log-in and teachers can use their professional log-in. Everyone has a log-in through their employer. Interview 12, 72-73 The learning paths mentioned also enable teachers to make minor adjustments and to create their own learning paths from OER materials, which they can in turn share more easily.
AM: Say you, you delete some part of the part. Okay, and you add some text in the start to think about these in your own context. And then it's much like a Spotify playlist. It's not scary to share a playlist because it is not your own work it's only your own list.
AW: You can make play or you can make some personal, like you can write your own text a little bit to personalize in some ways… Licencing issues are barriers of sharing content as OER.
AM: Yeah and add a YouTube video and another re-source… AW: Yeah, but if you only write some things and most of the other things are… other peoples that you just… that make a list, it's like a comment on the list in some ways, so… AM: That's not so scary I don't think about licensing because everything is already licensed and you just remix and add some comments. Interview 4, 287-291 Another way to encourage teachers to share is through in-school workshops with teachers.
The largest success that we have had in that area has been with bigger programs funded by the government, where they said 'ok, now, all schools should send teach ers on courses and they should make something in the workshops regarding to how they can teach, for instance, vocational students about subjects which is not so vocational. And we will say to the government 'oh, what a very good idea, but you want to have a portal where they have to share what they're doing at this workshops.' 'Oh good idea.' And we will make this portal for them and they will go to that, and all schools in Norway are sending teachers to these courses and they must produce something and put this on the portal. So, that is kind of sharing but it is like with the gun to the head, but they are sharing. Experience from other platforms such as Wikipedia shows that white men are more likely to be volunteers than other groups of people. It should therefore be considered to what extent voluntary participation should be limited or whether other forms of involvement should be retained (day-by-day secondment, project-related additional payment).
With regard to process development it should be noted that this is a result of the last 10 years and that the NDLA did not kick off with the perfect solution in many areas (e.g. software development), but therefore was able to deliver a product that schools could work with in the given time period. This situation secured further funding from the counties. Underestimation of the complexity of the development of a technological platform led to the fact that the solution that was eventually developed did not correspond to the requirements. However, over those ten years, the users collected considerable expertise, what requirement are essential for a large-scale OER platform. According to one interviewee, an online production environment for high-quality OER content with interactive content and no Flash, with community-supported quality assurance and a learner-friendly edition for the students, cannot be developed in anything less than a year.
As an organisation that has grown over the years, it has also been necessary to rethink the processes and structures that have gradually emerged. As a result, NDLA's structure has been redesigned in recent years and is now based on the philosophy of being lean.
What we have actually done is we had quite a lot of work in implementing lean/ lean six sigma similar management philosophy and tools into organizational practice. A lot of those projects aren't very successful.
Motivating teachers to share content is not an easy task. User-friendly structures and systematic recognition (e.g. over days off or in assessments) could help.
Where I have been working with those projects in the implementation, a lot in the implementation. So I knew that if we are going to have a hope in lean-orientating and that means end user-orientating all of NDLA. Then we have to start with the people looking in our organization. We had to start building competencies and that's how it started. We started sketching out some basics on who is responsible for what and in that we started the translation and we also involved all key personal in our organization in this transition. I think now that we can say that two years after, 18 months after it looks like we succeeded. We go to the core in lean, we started implementing, they would call it the 'Shoeheart-cycle' into businesses in Japan. The 'Shoeheart-cycle' is rather simple. It says plan what you are going to do, do what you planned, study the difference and act on the difference -either by changing your plan or better following your plan.
[…] Interview 3, 5-5 One effect of working as a virtual organisation is that NDLA is spread all over Norway. The work is done via Skype, hangouts or telephone. Through the described structures, decisions and questions are communicated and discussed in weekly meetings. At the beginning of each week there are team meetings, followed by a meet ing of the team leaders.
[…] This is the basic theory and what we do is, we do use that cycle on three levels: We use it operatively in the operative teams, we use it on tactics level on our leadership level (with my leadership team) and we do it on highest level. And that makes it possible for us to say when there is a plan from top we need to communicate to lower level, then it goes through me, then I have to find out what it means and together with my team implement that plan and detail it. And then my team managers, the team leaders, will take that into their management cycle. And in the same government structure we report back. So if, let's say, those people working with communicating with our users find out that 'well, this plan didn't work' and then report back 'this doesn't work' and our then our users tell us 'if you do a little bit adjustment here and there then it might work'. Then that message can go up to the next leadership level and back and also to the top. So it is a very simple structure and I'm actually working on the lean philosophy, it's day by day some competences such since we worked on on some education of my personal that it actually is follow up. Every week we have the same structure on the meetings, we have the same follow up, we distinct in separating decisions from actions from activity and so on. So we do lean, we also, for innovative work we do, we do lean start-up oriented methods in how we organize it and we also are inspired by design-thinking as a method. Interview 3, 5-5 With regard to the structuring processes, it was noted that from the outset the scope for creative minds must be sufficiently large in order to unfold and to develop a common vision and identity. However, this means that not every development will be a success, and the beginning can be more chaotic than process developers imagine. The structure of NDLA consists of a team comprising several small teams with weekly online meetings and a fixed structure in the leadership team and the respective working groups.
you let those committed souls give them some leverage or give them some play, give them a playground. You know, if you have a group of teachers who really want to do this, so let them do, let them follow their dreams in a way, give them credits for what they want to do and give them sort of loose chain, they can go quite far. I think it's important that you base the identity through a set of core values that has to do with sharing, openness, respect, tolerance, variety if you like, always keeping the students in your mind when you do things. So, developing some sort of core identity that does not inflict too much damage on each individuals, sort of dreams of creativity. You have to sort of let them bloom. And accept 'okay, oh you're so good with using films and teaching, develop it, use it, use you're great at it, you know? So, I think that's very important when you build an organization of those who come in here, they come in and then they can use themselves and they use what they are good at and get acknowledgement for that. And then gradually, I have to build sort of professionalism into it and more discipline. We start out with sort of letting the enthusiasts bloom. I think that is important. Interview 12, 121-121 This also means making the talents of the individual team members visible.
Perhaps, I think you need to appreciate the people who become part of the initiative, that they feel appreci ated, they have talents; and that you combine that with creating that identity of ‚we're on a mission, we're going to create something together'. Because it's easy to, when you start out the new..., you want to create this identity, that you narrow it down to some special methodology, special method: 'We are going to do it like this, something like that'. I think it is important to have that creativity and appreciation of the early enthusiasts. You need to make sure, it has to grow roots.

I: Don't say: This is way to go.
A: That's very dangerous. And that's what Øivind has been very good at as well. To appreciate the talents of each and one and at the same time keeping that vision alive, that everyone can agree upon. I think it is a balancing act because you would like to say, 'oh no, every one has to do this and output will become much better'. But you have to have blows in the organization with you and they have to feel appreciated and that they're part of something important. You have to find the right type of leader I think it's important. Interview 12, 177-179 Developing a team identity and fostering cohesion among team members who share a common vision and are convinced they are doing the right thing is particularly important in OER when dealing with hostility from publishers and the media.
A: So we have a lot of fun, actually, great fun but it's also hard when everybody hates you. What we do is not media-strategics, we're not very good with selling stories about how good we are and our strategy has been to 'OK, let's just work as fast as we can with changing the world. We have the time, they are not stopping us. Don't stop, start, don't stop. If they hate you, just continue.' And that has been our strategy, actually. If we have bad media, just continue. We can answer them and say what we think but we don't stop. Never stop, because the publishers have really tried so many times and the momentum is also vulnerable […] Interview 1, 145-145 NDLA has succeeded in implementing the approach of a multi-professional team. The pedagogical expertise of the teaching staff is complemented by that of product developers, photographers, media and web de-Appreciating the talents of individuals while pursuing a shared vision is, for an OER platform, a balancing act central to shaping the identity of individuals as part of a whole.
signers, computer scientists, lawyers and administrators. Through constant cooperation, the participants adopt procedures from other professions and thus build up their own expertise.

A:
At least what I think people should be aware and that still puzzles me and is a surprise for me, I mean I am a language teacher, I've studied German and English and Russian and Norwegian, I am just an ordinary teach er and now I sort of work closely with people who actually do web design and technology and it's, even though half the time I think I don't know what I'm doing just believe that you actually can do things and there will always be experts and you learn so many things. I mean just everyday think that there is so much to learn and don't be afraid that you don't how to do it or you're not educated for it but because of that work ing as a team, I mean, you will find out and a year ago I would have no idea on making and creating the platform that we made. I wouldn't know about too much anyway about user experience or testing or well, couple of years ago anyway.
You can learn new things and as long as you work together in a team, I think everything's possible. So, it's sometimes I think that 'okay, I'm just an ordinary teacher', and I am, but at least you can learn as you go, as well. So, I think that's just a team working together because I don't think without others you're nothing. You can't just think that you know everything yourself, so that's sort of also says it, when I spoke about the technology and using other peoples' technology. But just working as a team, I think that's a very good thing. Interview 11, 85-86 Ultimately, all professions are of decisive importance for the functioning of the entire organisation. Individual professionals may not be central, but they are particularly helpful for such an OER platform. For example, the visual level can often be covered with the help of a photographer who teaches media design at school and is seconded to NDLA in a half-time position.
In addition, there is the question of how the team is composed. Here, too, the topic of diversity was introduced by the interviewees. On the one hand, this refers to the aspect of cultural diversity, which is not particularly pronounced in the team.
On the other hand, the question of gender equality must be addressed. There seems to be an imbalance here, especially at the management level: I: Yeah, so how is it with gender? You already mentioned the white male Wikipedia-guys, is it in NDLA, is it equally distributed or is it more on the contentproducer-side? Is it women and men equally or is it more men or? Or more women?
A: Yeah, it's an interesting question. When we start back in I think this was quite equal. At this time, the top leaders in NDLA are men and the work is on the lowest levels almost everyone a woman, so I addressed this. Not everyone liked it, but at least some did (laughter). It's a danger. OK, it's like the rest of the society when you get to the leaders, there is more men and that shouldn't be like that so this I'm telling this young men, Øivind and Pål and so I say 'You have to choose not the people that are like you, you have to choose some other' but that's a problem for all of us. So I think we have to be aware of this. I'm so old that I see there are not so many years before I will be in pension, but I will strongly tell them, that this is really important. Interview 13, 71-72 As an OER platform: either develop a media strategy or a thick skin.
A multi professional team is needed for an OER platform in order to professionalize operations.
Also, the question of age diversity in the team has to be considered in order to be able to tap experiences of recent decades, on the one hand, and to not let the age span to the pupils become too big on the other.
The question of age diversity in the team is directly followed by the question of the transfer of organisational knowledge. It was an interesting idea that a person can only move up the career ladder in a company once it has been ensured that someone else can take over the previous tasks. Such a procedure also makes sense for an organisation like NDLA.
The issue of gender equity in the team should also be considered in the leadership of NDLA.

International cooperation
As shown in several contexts, it is very easy to build on NDLA's work. Content (texts, images, videos or multimedia content) and software (h5p, the new NDLA platform) can be used under the free licence. Even the structures can be used as a basis for national solutions. Here, too, Norway can benefit from the investments of other countries, as they can use the content if necessary, and the translation of content is more cost-effective than redesigning multimedia offerings themselves.
A study by the PEW Research Center shows that in 2018 2.5 billion people will own a smartphone. The worldwide, cost-free and unrestricted accessibility of educational content via the Internet also supports the democratisation of educational processes. In this sense, free access is seen as a step towards the implementation of the human right to education: A: And it's about ethics also. Because if learning is a human right and the best learning, or at least good learning, is a human right, how could we stop it? How can publishers say no because of more money? 'You have to pay me for learning, you have to pay me for your water, you have to pay me for the medicine!' And it's the same question basically, because when three-year-old in Africa don't get their Aids-medicine because it cost 15000 kroner a dose instead of being free, instead the production cost, why should child die? To fill your pocket? And it's actually about ethics and what kind of world do we want to build here? And I think it is basically the same question, it is not that obvious in the learning industry, but it is basically the same question. Why should the publisher be able The interfaces (APIs) of NDLA's new platform facilitate content distribution worldwide.
to take 15000 kroner for the learning resources for a year, which you won't be able to access the rest of your life, from one pupil? Interview 3, 86-86 Via the interfaces of the new platform, NDLA is able to automatically share the content with the world. If other platforms adopt these standards, the limited resources available can be used efficiently.
NDLA also supports other international projects such as the Global Digital Library (https://digitallibrary.io/).
In the NDLA team, international cooperation is seen as key to the global implementation of the human right to education.
With regard to future perspectives, three central areas were addressed. These include dealing with innovations, the new platform and the expansion of NDLA, including the creation of new OER platforms for other age groups.
With regard to the handling of innovation, there is the accusation that NDLA hinders innovation.
The publishers will say that we are killing innovation. I think Norway is the only place in Europe or in the world where people can get away with saying things like "OER stands against innovation." Politicians will think that open licensing is against innovation, they say that small edtech-companies, which they call them, they suffer because we don't buy from them, you will visit two of them, from which we buy from, we have limited amount of budget, so we can't use all of our budget on procurement. Interview 1, 129-129 It should be clearly pointed out that NDLA only receives 20% of the funds allocated and that 70% of these are effectively made available to the market. In light of this, 94% of the total money allocated is still available to the market.
Furthermore, NDLA's close connections to teachers and students as end users enable NDLA to research and describe their needs and thus promote innovation. The examples of Cerpus and Joubel (h5p) show that NDLA promotes innovation that subsequently spreads all over the world. Without the support of NDLA, h5p would not exist as an open standard for multimedia educational content.
The use of free licences for the content allows these two (and all other) companies to build on the success of NDLA. The creation of a new platform based on the experience of the last ten years is a collection of innovations implemented by an edtech company.
There is a four-stage decision-making process for deciding whether a technology or innovation is adopted by NDLA: This is purely organized around lean-startup and design thinking. So we have four decision points, decision point zero to four. And one idea in here, or ten ideas in here, develops to one to two ideas here, going into here. And what we do is trying to find out as much as we can about the idea as early as possible using as little money as possible. That means speaking to the users often. So, before we have an idea here, which is accepted for going through this process, we demand four questions answered. First question is: What's the need? Who owns this need? And if it is not the pupil and not the teacher, then beware. Why should NDLA do this? What makes us so special, so good? And is it within our business area to do it? And the last question is: Who can we test it on?
When those four questions are answered then you say 'this idea here, you can start looking at it'. Interview 3, 40-40 Such an approach prevents following every cost-intensive trend the entertainment industry tries to establish on the market (Augmented Reality, Virtual Reality, Gamification, 3D, Smart Home, Robots etc.). Individual offers (e.g. in the area of VR) are created by enthusiastic employees and can thus be tried out.
So far, ideas for innovations come from the team and only unsystematically from the outside, but there will be interfaces to students and teachers in the future.
NDLA supports innovations and smaller ed-tech companies like h5p and provides open standards for sustainability and free-to-use content.
Another approach of the cooperation partner Cerpus is to involve pupils themselves in the development of games and to enable them to develop questions on the topics dealt with. Formulating complex questions on a topic requires a deeper understanding of the subject matter than answering questions.
NDLA's new technological platform, which has already been mentioned a few times, is one of the concrete prospects for the future and raises great hopes among all NDLA interviewees.
Overall, NDLA's success will in future depend on the usability, stability and scalability of the new platform, as the existing solution with Drupal 6 is no longer sufficient to deal with the increasing number of users and materials.
For the editors of the subjects, the new platform means a considerable effort, but also the opportunity to further develop the subjects by foregoing the complete adoption of all content and, instead, only taking the more successful and current materials or to update others.
I: So actually, you would build a new platform and not just copy everything into the new platform, but… A: No…, yes, that's right. I would perhaps look at how much it has actually been used. And say that, all our resources that have been used that and that much, related to the number of students of course, will be brought along and fixed, so that will perhaps be 10 percent of our resources. So, I would focus on those, revise those, but not the whole 100 percent. Interview 12, 60-61 A further question concerns the age groups NDLA or another comparable government platform can offer content. On the one hand, it should be pointed out that NDLA is already being used -outside the actual target group -by pupils at the end of lower secondary level and by students at the beginning of their studies. On the other hand, the differences in political responsibility mean that it is not so easy to build on NDLA's positive experiences in primary and lower secondary education. In this case not the 18 counties are the politically responsible institutions, but the 400 individual municipalities. An expansion to primary and lower secondary level would be worthwhile, despite the higher expenditure due to the significantly higher number of pupils.
The idea of Distance Learning Centres could be of particular interest for vocational training, where pupils can learn together in a pedagogical setting in a mixture of specialist input online and social cooperation on site.
I think perhaps, that web teachers or net teachers or net/web guides also will play a role, because they are more available. And so.. I don't know. But in Norway, we have lots of hills and mountains, lots of distance. If you go to a place like Sykkylven, it's very hard to imagine, that you will have teachers for a huge variety of vocational subjects.
Well, let's say, you had a learning center, with pedagogues, counsellors, janitors, and you had sort of auditoriums or something, you had a lot of facilities for web-based teaching. Then you could sit and do a lot of the teaching through the web. And then you can have oil industry-related vocational subjects located in Stavanger. They could broadcast, or you can sort of like…, you could download it whenever you need it, here's the lecture showing you how to... And you could also imagine that students actually do things, they can operate on a frog two thousand miles away, or they can fix something in the car. You know.., you can imagine a lot of… […] The potential of primary and lower secondary schools (grades 1-10) for OER in Norway (and other countries) is even bigger because the number of students is even higher.
But there are lots of other things that you can sort of do… through the web. So, I think we have to…, what's a problem is…, have a place where people get together, so that you will have our social needs met and you have to… One of the most important things we teach our students is how to behave, how to recognize other people's skills, how to differentiate between a subject you're discussing and the person. So there are lots of competences that need to be learned. That can only be learned in a group. We need to see each other and shake hands and everything. In these learning centers you can have lots of topics taught in regular classrooms as well. More sort of traditional subjects, so... Interview 12, 92-96 Another attractive future perspective that was addressed in the interviews was the Brainbank concept developed by Cerpus. The aim here is not merely to document the contents learned in a learning diary, but also to connect them with each other by means of descriptions in one's own words and to create and visualise references to other contents already learned.
In the sense of a constructivist approach, at the end of the lesson, learners reflect on the most important contents and link them to at least one existing content. It would be exciting to combine such a constructivist approach with NDLA and other OER sources.
NDLA has its own innovation processes to test new technologies, to check their meaningfulness and to use as few resources as possible.

Conclusion
In summary, NDLA is a vivid example for many countries of how a state-funded OER platform can be successfully implemented. This study highlights opportunities and challenges and for the first time, provides a comprehensive insight into the Norwegian situation. The interviewees are NDLA employees as well as cooperation partners and opponents of NDLA.
With NDLA's free software and content licences, other countries do not have to start from scratch, but only adapt the system to their needs. As educational institutions around the world face similar challenges, cooperative (further) development of content and structures can help to use available resources as efficiently as possible for the benefit of students. This applies in particular to countries such as Germany or France, which have enormous potential for developing their own OER platforms due to their large number of pupils. The diagram of the federal states of Germany illustrates the number of pupils per federal state (rounded figures based on KMK statistics (Standing Conference of the Ministers of Education and Cultural Affairs of the States in the Federal Republic of Germany, 2018)). It becomes clear that joint action on the part of several federal states would make sense. If one euro per pupil and per subject would be made available, 7 million euro would be available annually per subject for joint action involving all federal states.
Of these, 4.9 million would be returned to the market if the Norwegian model was followed.
The organisational structures presented here also provide an insight into how such an undertaking can be successfully implemented and which challenges need to be overcome.
The implementation of a legal basis following the Norwegian law (see Chapter 4) makes lawsuits brought by the publishers of schoolbooks less likely, as there are already relevant court decisions at European level.
For inclusive pedagogy, the approach of state-financed OER is particularly appealing, as it is possible to pro vide services that already take differentiation into account and, above all, offer teachers the freedom of scope to further adapt the content to the needs of their pupils.
When developing inclusive materials, it can be helpful to describe pupils with different special educational needs by means of personas and to keep these in mind during the development. The multi-professional editorial teams could also benefit from systematically including teachers with qualifications in special needs education. It is important that learning with different objectives is incorporated into a pedagogical platform as a connecting document and into the calls for tenders. On the one hand, the question as to the crux of the matter has to be clarified in a special way and on the other hand, the question of the relevance of the topic for the different pupils has to be considered. Further suggestions for the design of inclusive OER can be found in Müller (2016).
All in all, it can be said that thanks to the openness of Norwegian colleagues, other countries can now benefit in many ways from NDLA's experience and products.
Content and structures of new state run platforms can be built on the experience, software and content of NDLA.

Notes on methods
The present results are based on 14 expert interviews, which were conducted personally in Norway in March 2017.
In the run-up, possible contact partners were identified together with NDLA's CEO at the time, Øivind Høines. These include members of NDLA from the management level on the one hand and cooperation partners from companies working with NDLA and a representative of the schoolbook pub lishers, on the other. The interviews were based on a collection of questions and topics compiled in the run-up to the study, in each case clarifying with the interviewees which areas are of particular interest to them.
The attempt to open up research right from the start by not only asking one's own questions but also further questions from the community, can be considered as failed due to a lack of feedback. A mindmap with the collection of topics was distributed via the usual communication channels (Ebner et al., 2015, p.22).
The interviews were conducted in English, recorded and then transcribed. The transcription is based on Kuckartz, Dresing, Rädiker and Tefer (2007). The transcrip tion of the length of the pauses etc. was omitted.
The qualitative content analysis according to Mayring and Brunner (2009) was used for the analysis. An inductive procedure for the formation of categories made it possible to focus on what the interviewees said and to maintain the necessary openness. MAXQDA was used to create a category system, whereby each category was provided with a description and an exemplary anchor example.
The selected transcribed interview excerpts were approved for publication by the interviewees. For legal reasons, no permission was granted for one of the interviews.

Systematic dissemination
For communication of the research results, the present publication has chosen an approach of systematic dissem ination, which is not necessarily usual in science communication, at least not in the field of education.
The relevant stakeholders in education policy (education policy spokespersons of the parties in the federal states and the respective parliamentary groups), school administrations, state institutes and foundations were identified by means of queries to the institutions and a snowball procedure. The postal addresses of these persons were recorded and the German version of this publication was sent to these stakeholders by mail. All this was done in hope of achieving a higher degree of reception than only publishing the results in an academic journal in English. At the same time, it was published as an Open Access version and was disseminated via the community-specific channels in German and English.

Acknowlegements
First and foremost I would like to thank the interviewees from the NDLA team as well as the partners of Cerpus, Joubel, "Kommune Forlaget" and the Norwegian Association of Textbook Publishers for their participation in the interviews.
The presented project, the present Open Access publication and the conference based on it in November 2019 were made possible by the research funding of the University of Bremen as well as the support of the Municipality of Bremerhaven, the Hanse-Wissenschaftskol-leg, the Max Traeger Foundation and Faculty 12: Educational Sciences of the University of Bremen.
Additional thanks go to Ole Nekarda, Leanne Lütjen, Lea Fischer, Anna Driver, Anna Zimmermann, Dan Smith (from language-associates.de), Ghislaine Wijsman and Sandra Stockmann for their support in the work on this project. Special thanks go to Janine Biermann for the design, Gabi Fahrenkrog, Jöran Muuß-Merholz and the publishing house ZLL21.