Hi,

I think that Stephane's points are valid, however I still do not like solution (a). Solution (b) may work however, as Michel points out, it won't have a link between the files that are used commonly across future repositories.

I have looked into this a bit more, and am now in favor of using subtrees as opposed to submodules for the code that we're spinning off (git subtree is standard in git 1.8, so you may need to update your git from wheezy-backports if you're using wheezy, where the git version is 1.7).

Manual:
https://github.com/apenwarr/git-subtree/blob/master/git-subtree.txt

Tutorials:
https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844
http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree

We should do some testing before committing to this solution, but my first test seems to work well. Essentially, we can use Stephane's solution (b) to do exactly what we want and use subtrees to take care of common code. Here's an outline of the solution I see:

* create dates, dseries, and reporting repositories, with master branches (the standalone-unstable code), versioned branches (the standalone-stable code, which we may never actually need....it depends on if these projects every get lives of their own), and a dynare branch When we include these repositories in dynare, we will point to the dynare branch. To create each of these repositories, we will use the git subtree split command, to retain their development histories.

* in the dates, dseries and reporting repositories, include dynare.git as a subtree. We remove all files that are not necessary for that individual repository. For example, I don't think we'll need a dynare subtree at all for reporting, whereas we'll need it for dseries (for the load_csv functions, name2tex, etc.). I'm not sure about dates, but we'll see with testing.

* when these files are updated in Dynare, we need to do a: git subtree pull from within every repository that includes dynare as a subtree. This will merge all new changes in dynare into each repository that contains it as a subtree. This means that we won't have to remove files that we have already removed from the repository and won't have to worry about changes to files that have been removed. But, we'll have to remove any new files that have been added. This is added work, but so is git, and I think it's worth it to have have code that runs independently. Also, it's the only way I see of keeping track of changes in dynare without creating a "cepremap_toolbox" repository.

* on the dynare branch of each of these repositories, we keep only the code that is in dynare-unstable (i.e. not the subtrees that they contain or the auxiliary files like README.md or setup scripts). This has the added benefit of allowing us to edit these files directly in dynare and do a: git commit; git subtree push directly from dynare.git to update both dynare and the subtree repository. We will then have to go to these repositories and cherry-pick the changes to the master (standalone-unstable) branch and any versioned branches (if they exist) if it is a bug fix.

I think this solves all of the problems we had been talking about. Is this ok with everyone?

Best,
Houtan

On 02/11/2015 02:56 PM, Stéphane Adjemian wrote:

Hi,

I do not think that it makes sense to put all the modules in one git repository. I do not perceive the advantage with respect to the previous situation (where all the codes were in the dynare git repository).Here are some arguments

(1) We (with Fred) already started to put particles in a separate git repository (it’s not in master yet). The reason for this move is that we believe that this code may be used for projects unrelated with DSGE or with Dynare. I do not understand why we should put this module in a common git repository, since this project has nothing in common with dates, dseries, …

(2) For me, the partition between different git repositories should not be a matter of size but of logic. If I want to use the dates tools I do not need to install the reporting tools or the particle filter codes. For instance, if I want to run a test suite specific to the dates module on a service like travis-ci, I just need to clone the dates and tests modules.

(3) By putting all the projects in a single git repository we loose some flexibility. For instance, I think that it would be no more possible to point to different branches/tags for the different submodules in Dynare (which is the case currently).

If the concern of Houtan is the distribution of the reporting tools (which depend on dates and dseries) or the dseries module (which depends on dates), we can think of simpler solutions:

(a) We can provide a small matlab installation script that downloads a snapshot of the required modules (basic users do not need to clone the modules with history).

(b) We can use different branches. For instance dynare can point to the master branch of reporting git repository, and we may have also a branch called standalone which is the same than master but with dates and dseries as submodules.

I favor the first solution (because when we distribute code we do not distribute the entire history)

Best,
Stéphane.

Le 10 févr. 2015 à 09:49, Michel Juillard <michel.juillard@mjui.fr> a écrit :

Let see what Stephane says.

Houtan writes:

This sounds good to me.

Can you create the repository in Github? I'll take care of moving
everything over and making sure it works.

Best,
Houtan


On 02/09/2015 04:25 PM, Michel Juillard wrote:

After some thinking, my proposal is in fact to maintain a single GitHub
repository called cepremap_toolbox in lack of a better name, that would
contain, dates, dseries, reporting and utilities used both in Dynare and
in these projects.

Then to stop the separate GitHub repositories dates and deseries

The idea that for the moment it is not a huge imposition for somebody
who wants to use only our dates to download everything. If and when
cepremap_toolbox becomes too large we will think about a better
packaging.

Best

Michel


Stéphane Adjemian writes:

Hi,

I am not sure that there is any advantage in creating a new git repository (with dseries, dates, … as submodules) which would be (if I do not misunderstand michel’s proposition) a submodule in Dynare… But I am sure that this would add more work. If I add a new method in dates and want to use it in Dynare, I need to:

(1) update the dates repository
(2) update submodule dates in cepremap-tools repository
(3) update submodule cepremap-tools in dynare repository

I would prefer not to pay step (2) each time I need to update dates, dseries, particles, … I think that a submodule for all the common utilities in Dynare would be useful, but we must decide what we put in it. I probably didn’t push this change (and I can revert it) but I finally put load_*_file_data.m routines in dseries.

Best,
Stéphane.

Le 9 févr. 2015 à 09:04, Michel Juillard <michel.juillard@mjui.fr> a écrit :

I'm in favor of having dates and dseries and, maybe reporting, work independently from
Dynare.
In a first stage, I don't see it as a big drawback to have to download
separately dates and dseries. At a later stage, we may think about have
a more user friendly distribution.
The more delicate issue for which I don't have a simple solution is
those files upon which both dseries, reporting and Dynare rely upon:
check_file_extension.m
load_csv_file_data.m
name2tex.m
iscellofchar.m
etc.
The logic would be to have yet another submodule like
cepremap_utilities, but it seems a little farfetched.
I'm thinking as I'm writing: maybe a better approach is to start with a single module
cepremap_toolbox (or any other name showing its independance from
Dynare) with dates, dseries, reporting and common utilities and
only think about splitting it up when one part is growing too much.

Best

Michel

Houtan writes:

Hi....

On 01/28/2015 12:36 PM, Stéphane Adjemian wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

Answers below...

Le 26/01/2015 10:40, Houtan a écrit :

Hello,

If I should have any spare time this week, I wanted to work on
https://github.com/DynareTeam/dynare/issues/811.

Looking into this has brought up several questions. If I
understand correctly, the idea of moving dates and dseries into
their own repositories was that they can be used as standalone
programs. As such, we want to permit people who can benefit from
this code to use it directly instead of having to download Dynare
as a whole. In essence, we're treating them as libraries, which I
think is a good idea.

Initially the idea was to make these codes available without
downloading the whole Dynare. But I did not intend to produce
standalone programs. For me it's not a problem if the dseries package
requires another package (dates package in this case). I do not like
the idea of having dates as a submodule of dseries, or dates and
dseries as submodules of reporting, ... I do not see any advantage
here, only more work.

The advantage is to simplify the process for the end user, having
something that requires minimal setup. Anyway, as it stands now (though,
as you point out, we could make a script that does this), the end user
still needs to download dynare in order to have access to some of the
files that these submodules depend on. E.g.:
check_file_extension.m
load_csv_file_data.m
name2tex.m
iscellofchar.m
etc.

In fact, if the only advantage is to save the user from downloading the
12mb of dynare source code, and if you don't intend to make these
submodules work independently, I don't see the advantage of spinning
them off into submodules in the first place. It actually just serves to
make our code more convoluted. I think that would be worth it if we are
willing to put in the work to make the products (dates, dseries, and
reporting) completely independent. But, if not, it just doesn't seem to
make sense.

To be clear, I don't think it's much work to make these work independently.

We would have to include the files above in a "utilities" submodule,
that would include general matlab functions that are used in various
parts of dynare (the files that you have already placed in
matlab/utilities/general in addition to some others). Then, as I
specified before, include dates as a submodule in dseries, and dates and
dseries as submodules of reporting. It doesn't seem like too much work
at all. What is the problem with this?

Best,
Houtan

But, as it's setup now, dseries is not a standalone program. In
order to make it a standalone program, we'd need to add dates as a
submodule of this program and probably have a configuration script
that adds the dates repository to the Matlab/Octave path. At least,
that's how I'd like to set up reporting: add dates and dseries as
submodules to reporting so that someone could clone the reporting
repository, initialize its submodules and use it as a standalone
program.

We just need to explain that if one wants to use the reporting module,
then the dates and dseries modules must be in the Matlab/Octave's
path. We can also provide a script to install these modules (but we do
not need git to do that).

Doing so would not present a problem to Dynare users as Dynare
could still include a dates submodule, a dseries submodule that
contains a dates submodule, and a reporting submodule that contains
both a dseries and dates submodule. As long as the git clone or the
git submodule init are not issued with the --recursive switch, the
nested submodules will not be brought in and hence the code will
not be duplicated.

Furthermore, I think we should discuss the location of submodules
within Dynare. Currently, they're in several places:

1) contrib 2) matlab/utilities/tests 3) matlab/modules

This is confusing. Though I don't have a strong opinion as to the
logic we should impose on the submodules we include, I believe
there should be a logic. Should all external submodules be included
in contrib? should all internal matlab submodules be included in
matlab/modules? Should all matlab tests be included in
tests/modules (matlab/utilities/tests has to be changed, in my
opinion)? It would be nice if we could make a decision about the
organization of the submodules within Dynare as a group so that,
going forward, we have a coherent structure that does not evolve
organically but rather in an organized and logical way.

I plan to put the tests routines under matlab/modules. I think that we
should have only two locations. The logic being that some codes are
supported/produced by us, in modules folder for instance, and other
codes are not produced/supported by us (we do not ensure that these
codes will work in future versions of Dynare), in a contribs folder.

Best,
Stéphane.

Best, Houtan _______________________________________________ Dev
mailing list Dev@dynare.org
https://www.dynare.org/cgi-bin/mailman/listinfo/dev

- --
Stéphane Adjemian
Université du Maine, Dynare Team
-----BEGIN PGP SIGNATURE-----

iQQcBAEBCgAGBQJUyMnRAAoJEKbUTLnGTOd7jyEgAJqgY6Cx8Gt08a3fSLdb4coE
rHdkEFFp8/w6EeM4QOWuXB9DnmvYTActoOCWgWaU0apBJr74Pl0JvpOxT1zuQOt1
bm7s9sVZEXIUu6UoTjqXXQRsI+2g6NqFsz2jYB84eX2orNYQtYjicieg1zshKA21
iONZiyksAw8KaM5aO+z4sQ9wgz/oKOYvedtOqCwED8ZDhguAMEyUJO4db13EPYXd
pKr0SayW1y4E+ORriVKRHsO6w9HpIzHK2cPBZlihLWcJAKgl4ZVMQx89RpJBXhFK
jqfFWkvZuk/yaq/MgNfy+avKz+e4oqz/tDW3LKzfP17VVTVb+YRywRgbdycE6Nkr
YXu+Er9zaCWYMWwQiJzwj6/8OE2pWLFHoGfu4aIDpB52fuVfxlP9934IowxvoD4q
Sk13rPNwlOkbn/s35zuFflrs0cXEqcpedoQejudwYB7Nb5NGJRRN7UM2cfmuOUf+
GIDNRwPns4ve5RWhHRhtcyUy0asA+EZDgXMPBZl5fRL5hWbUyVGc6hdBHhxnPJ3V
RbFxqAIb1V8AUEbg5StdZzGBGX0FSWgYRDcETXgqa51nfMIrd1b0qUQASl0LpFBO
F+GGOHC+UPUnHnCv5LY+YqaMUuQzFq1a7+1B/0GB4x7UkJF7iCYMG1qmhCRJ47wI
qt67NalYOTK7qDwi3IO+JF5Nn+GwiBbuYMPzOurJ5TllJBryu65lLR2Ni4ahrcoO
TtU4PLnZeTpiWqWA6IK6XH8WjTCaCPwP553tbxeG6zafuX2XCIzdPzy2LIqSiaWl
lZ6MXLmV8roj4qD4IWXVPurZGWvlqs5OcbVwmYd6KltC+gosyHGEnvMuKjOw/wv4
fn6YzbWppQ37mRo0vkljmChQ4nOBQpvFbZvcBk/uDBTfwqH7poPxE06St4EeKHOV
huNxTNbSmkJXDf0kayrNL1TattBqiybAKCaIUCZpDagvzOrPPOgT/oZjVN9le/Hn
cGcR4NloeelR5UBeZvsPfewbnajGJia35eKmJlc1Db9fYzJgNjzyUvVeM1ORQL5k
G8HDZUhsTomvkQ5CGhjh3imO0JSDoPCy8JjpT8w+DuoLd0Ou+cY13Z/IKlyzF9E9
4lA7di8z7fMhmiS2c1+qa3OKOoPY5FmNmQBE6XP9zh7z/L0pHGAJ74ye5Iq3+VHy
6TgzLS+1qd1T+Ko+8CY8rHobgdbFrnFDZW5r1QwVwNfwe0S0ckfMTwP6zBo7RNlt
H/zOBu6qdkS8SpeBe+26PjowxvWm+1+TmeHZHA6hUsGaKvVFR3QxszGwkarPbseZ
pE8Qi/yOfGVLynmA6ylK1WINRvW5oVYsDzzWKBs1fHF9HsN+twGLtKheGh2ugrg=
=2Aq0
-----END PGP SIGNATURE-----
_______________________________________________
Dev mailing list
Dev@dynare.org
https://www.dynare.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
Dev@dynare.org
https://www.dynare.org/cgi-bin/mailman/listinfo/dev

--
Michel Juillard
_______________________________________________
Dev mailing list
Dev@dynare.org
https://www.dynare.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
Dev@dynare.org
https://www.dynare.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
Dev@dynare.org
https://www.dynare.org/cgi-bin/mailman/listinfo/dev

--
Michel Juillard
_______________________________________________
Dev mailing list
Dev@dynare.org
https://www.dynare.org/cgi-bin/mailman/listinfo/dev

_______________________________________________
Dev mailing list
Dev@dynare.org
https://www.dynare.org/cgi-bin/mailman/listinfo/dev