I don't know what you mean by "deals with sub-directories." Yes, it puts the files in a folder not the cwd, but I don't see how this is a problem.
In spinning off code into separate repositories we are, by definition, moving production out of dynare and thinking of them as separate entities. Dates is no longer dynare code. It's a separate project that is useful for dynare and, hopefully, for other projects. It should be an independent project, ideally with its own distribution cycle, master branch, and versioned release branches. That said, again, we may never move beyond a version 1 and the distribution cycle will mirror dynare's in practice. But, at least I think, it should be thought of and treated as an independent project.
As an independent project, it should be set up as such. It should standalone be distributed in a working fashion. As we don't want to weigh it down by having dynare as a submodule of dates, which would be overkill, we have two options (that I see):
1) include dynare as a subtree of dates, remove all unnecessary files (essentially, only keeping a handful of .m files). This will require us to update the subtree whenever one of these files is updated in dynare.
2) create a cepremap-toolbox that contains all .m files that are used by at least one dynare submodule (dates, dseries, reporting, or particle). Similarly here, we'll have to update the repository (and dynare) whenever cepremap-toolbox is updated.
1) is slightly less work as the changes to the files are made directly in dynare. Hence, we only need to do a git subtree pull from the master branch on dates.git to get the changes made in dynare. 2) would require us to update pointers in dates.git and dynare.git, after making a new commit in cepremap-toolbox.git. I'm fairly certain we can automate the updating process for both 1) and 2), so the amount of work shouldn't matter.
If we do 2), the structure is rather clear as the cepremap-toolbox would work the way our submodules work now. After having read the tutorial here ( https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 ) and worked with two test git repositories, I tend to prefer 1) as I think it's simpler and more elegant. But, both of these options will work fine.
This definitely complicates the structure of dynare, because we're creating N projects out of 1. That's the trade-off for having independent/standalone code, and I think what Stephane wanted to avoid.
I'm sorry if the mails I'm sending are not clear. Perhaps I can better explain this in person or over the phone when I'm back in Paris.
Best, Houtan
On 02/12/2015 10:46 AM, Michel Juillard wrote:
My impression was that subtree only deals with sub-directories but maybe I'm wrong. Even if subtree is very flexible, I think that we should keep designating a primary location where the various pieces of code are developped and that are then used elsewhere. I'm a bit worried of a model where, say, dseries is developped in GitHub dseries repository but it contains a subtree of utilities that are developped inside GitHub Dynare repository. I don't have the idea of having reciprocal dependencies between GitHub projects. We should have upstream and downstream projects otherwise we will always forget the layout.
Best
Michel
Houtan writes:
Hi Michel,
Why do you say that? With subtree, we can include any files we need from dynare in any repositories that we need. They can be updated after they are updated in dynare, and hence any bugs in these files can easily be incorporated into other repositories.
If I understood correctly, one difficulty we have in creating a cepremap-toolbox.git is that we don't have a logical idea of what to include and what to exclude. And, even if we did, we would still include this repository as a subtree/submodule in any other repository.
In fact, if the idea is to move these utilities out of dynare, I don't see where we would stop. Does the preprocessor become a submodule? What about the entire matlab folder? That could be split up into many submodules. Dynare++? Every mex folder its own repository? Maybe I wasn't involved in the discussions, so I don't know the logic driving the creation of submodules.
Best, Houtan
On 02/12/2015 10:10 AM, Michel Juillard wrote:
subtree looks cool and may be part of the answer. However, in itself, it doesn't solve the problem of the general utility functions used by dseries. We still need to find them a place outside of Dynare git
Best
Michel
Houtan writes:
Hi,
I think that Stephane's points are valid, however I still do not like solution (a). Solution (b) may work however, as Michel points out, it won't have a link between the files that are used commonly across future repositories.
I have looked into this a bit more, and am now in favor of using subtrees as opposed to submodules for the code that we're spinning off v (git subtree is standard in git 1.8, so you may need to update your git from wheezy-backports if you're using wheezy, where the git version is 1.7).
Manual: https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt
Tutorials: https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844 http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree
We should do some testing before committing to this solution, but my first test seems to work well. Essentially, we can use Stephane's solution (b) to do exactly what we want and use subtrees to take care of common code. Here's an outline of the solution I see:
- create dates, dseries, and reporting repositories, with master
branches (the standalone-unstable code), versioned branches (the standalone-stable code, which we may never actually need....it depends on if these projects every get lives of their own), and a dynare branch When we include these repositories in dynare, we will point to the dynare branch. To create each of these repositories, we will use the git subtree split command, to retain their development histories.
- in the dates, dseries and reporting repositories, include dynare.git
as a subtree. We remove all files that are not necessary for that individual repository. For example, I don't think we'll need a dynare subtree at all for reporting, whereas we'll need it for dseries (for the load_csv functions, name2tex, etc.). I'm not sure about dates, but we'll see with testing.
- when these files are updated in Dynare, we need to do a: git subtree
pull from within every repository that includes dynare as a subtree. This will merge all new changes in dynare into each repository that contains it as a subtree. This means that we won't have to remove files that we have already removed from the repository and won't have to worry about changes to files that have been removed. But, we'll have to remove any new files that have been added. This is added work, but so is git, and I think it's worth it to have have code that runs independently. Also, it's the only way I see of keeping track of changes in dynare without creating a "cepremap_toolbox" repository.
- on the dynare branch of each of these repositories, we keep only the
code that is in dynare-unstable (i.e. not the subtrees that they contain or the auxiliary files like README.md or setup scripts). This has the added benefit of allowing us to edit these files directly in dynare and do a: git commit; git subtree push directly from dynare.git to update both dynare and the subtree repository. We will then have to go to these repositories and cherry-pick the changes to the master (standalone-unstable) branch and any versioned branches (if they exist) if it is a bug fix.
I think this solves all of the problems we had been talking about. Is this ok with everyone?
Best, Houtan
On 02/11/2015 02:56 PM, Stéphane Adjemian wrote:
Hi,
I do not think that it makes sense to put all the modules in one git repository. I do not perceive the advantage with respect to the previous situation (where all the codes were in the dynare git repository).Here are some arguments
(1) We (with Fred) already started to put particles in a separate git repository (it’s not in master yet). The reason for this move is that we believe that this code may be used for projects unrelated with DSGE or with Dynare. I do not understand why we should put this module in a common git repository, since this project has nothing in common with dates, dseries, …
(2) For me, the partition between different git repositories should not be a matter of size but of logic. If I want to use the dates tools I do not need to install the reporting tools or the particle filter codes. For instance, if I want to run a test suite specific to the dates module on a service like travis-ci, I just need to clone the dates and tests modules.
(3) By putting all the projects in a single git repository we loose some flexibility. For instance, I think that it would be no more possible to point to different branches/tags for the different submodules in Dynare (which is the case currently).
If the concern of Houtan is the distribution of the reporting tools (which depend on dates and dseries) or the dseries module (which depends on dates), we can think of simpler solutions:
(a) We can provide a small matlab installation script that downloads a snapshot of the required modules (basic users do not need to clone the modules with history).
(b) We can use different branches. For instance dynare can point to the master branch of reporting git repository, and we may have also a branch called standalone which is the same than master but with dates and dseries as submodules.
I favor the first solution (because when we distribute code we do not distribute the entire history)
Best, Stéphane.
Le 10 févr. 2015 à 09:49, Michel Juillard michel.juillard@mjui.fr a écrit :
Let see what Stephane says.
Houtan writes:
> This sounds good to me. > > Can you create the repository in Github? I'll take care of moving > everything over and making sure it works. > > Best, > Houtan > > > On 02/09/2015 04:25 PM, Michel Juillard wrote: >> After some thinking, my proposal is in fact to maintain a single GitHub >> repository called cepremap_toolbox in lack of a better name, that would >> contain, dates, dseries, reporting and utilities used both in Dynare and >> in these projects. >> >> Then to stop the separate GitHub repositories dates and deseries >> >> The idea that for the moment it is not a huge imposition for somebody >> who wants to use only our dates to download everything. If and when >> cepremap_toolbox becomes too large we will think about a better >> packaging. >> >> Best >> >> Michel >> >> >> Stéphane Adjemian writes: >> >>> Hi, >>> >>> I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to: >>> >>> (1) update the dates repository >>> (2) update submodule dates in cepremap-tools repository >>> (3) update submodule cepremap-tools in dynare repository >>> >>> I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries. >>> >>> Best, >>> Stéphane. >>> >>> >>>> Le 9 févr. 2015 à 09:04, Michel Juillard michel.juillard@mjui.fr a écrit : >>>> >>>> I'm in favor of having dates and dseries and, maybe reporting, work independently from >>>> Dynare. >>>> In a first stage, I don't see it as a big drawback to have to download >>>> separately dates and dseries. At a later stage, we may think about have >>>> a more user friendly distribution. >>>> The more delicate issue for which I don't have a simple solution is >>>> those files upon which both dseries, reporting and Dynare rely upon: >>>> check_file_extension.m >>>> load_csv_file_data.m >>>> name2tex.m >>>> iscellofchar.m >>>> etc. >>>> The logic would be to have yet another submodule like >>>> cepremap_utilities, but it seems a little farfetched. >>>> I'm thinking as I'm writing: maybe a better approach is to start with a single module >>>> cepremap_toolbox (or any other name showing its independance from >>>> Dynare) with dates, dseries, reporting and common utilities and >>>> only think about splitting it up when one part is growing too much. >>>> >>>> Best >>>> >>>> Michel >>>> >>>> Houtan writes: >>>> >>>>> Hi.... >>>>> >>>>> On 01/28/2015 12:36 PM, Stéphane Adjemian wrote: >>>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>>> Hash: SHA512 >>>>>> >>>>>> Hi, >>>>>> >>>>>> Answers below... >>>>>> >>>>>> Le 26/01/2015 10:40, Houtan a écrit : >>>>>>> Hello, >>>>>>> >>>>>>> If I should have any spare time this week, I wanted to work on >>>>>>> https://github.com/DynareTeam/dynare/issues/811. >>>>>>> >>>>>>> Looking into this has brought up several questions. If I >>>>>>> understand correctly, the idea of moving dates and dseries into >>>>>>> their own repositories was that they can be used as standalone >>>>>>> programs. As such, we want to permit people who can benefit from >>>>>>> this code to use it directly instead of having to download Dynare >>>>>>> as a whole. In essence, we're treating them as libraries, which I >>>>>>> think is a good idea. >>>>>> Initially the idea was to make these codes available without >>>>>> downloading the whole Dynare. But I did not intend to produce >>>>>> standalone programs. For me it's not a problem if the dseries package >>>>>> requires another package (dates package in this case). I do not like >>>>>> the idea of having dates as a submodule of dseries, or dates and >>>>>> dseries as submodules of reporting, ... I do not see any advantage >>>>>> here, only more work. >>>>> The advantage is to simplify the process for the end user, having >>>>> something that requires minimal setup. Anyway, as it stands now (though, >>>>> as you point out, we could make a script that does this), the end user >>>>> still needs to download dynare in order to have access to some of the >>>>> files that these submodules depend on. E.g.: >>>>> check_file_extension.m >>>>> load_csv_file_data.m >>>>> name2tex.m >>>>> iscellofchar.m >>>>> etc. >>>>> >>>>> In fact, if the only advantage is to save the user from downloading the >>>>> 12mb of dynare source code, and if you don't intend to make these >>>>> submodules work independently, I don't see the advantage of spinning >>>>> them off into submodules in the first place. It actually just serves to >>>>> make our code more convoluted. I think that would be worth it if we are >>>>> willing to put in the work to make the products (dates, dseries, and >>>>> reporting) completely independent. But, if not, it just doesn't seem to >>>>> make sense. >>>>> >>>>> To be clear, I don't think it's much work to make these work independently. >>>>> >>>>> We would have to include the files above in a "utilities" submodule, >>>>> that would include general matlab functions that are used in various >>>>> parts of dynare (the files that you have already placed in >>>>> matlab/utilities/general in addition to some others). Then, as I >>>>> specified before, include dates as a submodule in dseries, and dates and >>>>> dseries as submodules of reporting. It doesn't seem like too much work >>>>> at all. What is the problem with this? >>>>> >>>>> Best, >>>>> Houtan >>>>> >>>>>>> But, as it's setup now, dseries is not a standalone program. In >>>>>>> order to make it a standalone program, we'd need to add dates as a >>>>>>> submodule of this program and probably have a configuration script >>>>>>> that adds the dates repository to the Matlab/Octave path. At least, >>>>>>> that's how I'd like to set up reporting: add dates and dseries as >>>>>>> submodules to reporting so that someone could clone the reporting >>>>>>> repository, initialize its submodules and use it as a standalone >>>>>>> program. >>>>>> We just need to explain that if one wants to use the reporting module, >>>>>> then the dates and dseries modules must be in the Matlab/Octave's >>>>>> path. We can also provide a script to install these modules (but we do >>>>>> not need git to do that). >>>>>> >>>>>>> Doing so would not present a problem to Dynare users as Dynare >>>>>>> could still include a dates submodule, a dseries submodule that >>>>>>> contains a dates submodule, and a reporting submodule that contains >>>>>>> both a dseries and dates submodule. As long as the git clone or the >>>>>>> git submodule init are not issued with the --recursive switch, the >>>>>>> nested submodules will not be brought in and hence the code will >>>>>>> not be duplicated. >>>>>>> >>>>>>> Furthermore, I think we should discuss the location of submodules >>>>>>> within Dynare. Currently, they're in several places: >>>>>>> >>>>>>> 1) contrib 2) matlab/utilities/tests 3) matlab/modules >>>>>>> >>>>>>> This is confusing. Though I don't have a strong opinion as to the >>>>>>> logic we should impose on the submodules we include, I believe >>>>>>> there should be a logic. Should all external submodules be included >>>>>>> in contrib? should all internal matlab submodules be included in >>>>>>> matlab/modules? Should all matlab tests be included in >>>>>>> tests/modules (matlab/utilities/tests has to be changed, in my >>>>>>> opinion)? It would be nice if we could make a decision about the >>>>>>> organization of the submodules within Dynare as a group so that, >>>>>>> going forward, we have a coherent structure that does not evolve >>>>>>> organically but rather in an organized and logical way. >>>>>> I plan to put the tests routines under matlab/modules. I think that we >>>>>> should have only two locations. The logic being that some codes are >>>>>> supported/produced by us, in modules folder for instance, and other >>>>>> codes are not produced/supported by us (we do not ensure that these >>>>>> codes will work in future versions of Dynare), in a contribs folder. >>>>>> >>>>>> Best, >>>>>> Stéphane. >>>>>> >>>>>> >>>>>> >>>>>>> Best, Houtan _______________________________________________ Dev >>>>>>> mailing list Dev@dynare.org >>>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>> - -- >>>>>> Stéphane Adjemian >>>>>> Université du Maine, Dynare Team >>>>>> -----BEGIN PGP SIGNATURE----- >>>>>> >>>>>> iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE >>>>>> rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1 >>>>>> bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21 >>>>>> iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd >>>>>> pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK >>>>>> jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr >>>>>> YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q >>>>>> Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+ >>>>>> GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V >>>>>> RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO >>>>>> F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI >>>>>> qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO >>>>>> TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl >>>>>> lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4 >>>>>> fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV >>>>>> huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn >>>>>> cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k >>>>>> G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9 >>>>>> 4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy >>>>>> 6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt >>>>>> H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ >>>>>> pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg= >>>>>> =2Aq0 >>>>>> -----END PGP SIGNATURE----- >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> Dev@dynare.org >>>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> Dev@dynare.org >>>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>>> -- >>>> Michel Juillard >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@dynare.org >>>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev >>> _______________________________________________ >>> Dev mailing list >>> Dev@dynare.org >>> https://www.dynare.org/cgi-bin/mailman/listinfo/dev > _______________________________________________ > Dev mailing list > Dev@dynare.org
> https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Michel Juillard _______________________________________________ Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev
Dev mailing list Dev@dynare.org https://www.dynare.org/cgi-bin/mailman/listinfo/dev