Opened 5 years ago

Last modified 3 years ago

#3700 reopened enhancement

Copyright years update should be automated

Reported by: tomek Owned by: UnAssigned
Priority: medium Milestone: Outstanding Tasks
Component: build system Version: git
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: DHCP Feature Depending on Ticket:
Estimated Difficulty: 0 Add Hours to Ticket: 0
Total Hours: 2 Internal?: no

Description (last modified by tomek)

Updating all copyright years in every file we change is tedious.
We should borrow the scripts from BIND9 that do that automatically.

Subtickets

Change History (21)

comment:1 Changed 5 years ago by hschempf

  • Milestone changed from Kea-proposed to DHCP Outstanding Tasks

comment:2 Changed 4 years ago by tomek

  • Milestone changed from DHCP Outstanding Tasks to Outstanding Tasks

Milestone renamed

comment:3 Changed 4 years ago by tomek

  • Description modified (diff)
  • Milestone changed from Outstanding Tasks to Kea1.1

Updating the years manually is tedious. Let's automate this in early 2016.

comment:4 Changed 4 years ago by fdupont

  • Owner set to fdupont
  • Status changed from new to accepted

Copy and adapt bind9 scripts (which BTW detect copyrights to update but don't update them).

comment:5 Changed 4 years ago by fdupont

Almost done. There are 2 scripts and a copyrights file with all regular file names (git ls-files) and either an "X" or the last modification year. The first script checks if this file is up to date, the second rebuilds it from the content of the current repo. The idea is a diff shows what to be fixed.
In fact most of the remaining work is to write a documentation...

comment:6 Changed 4 years ago by fdupont

  • Owner changed from fdupont to UnAssigned
  • Status changed from accepted to reviewing

Ready for review. If the ChangeLog entry should not be very verbose I propose "Added two scripts to detect ISC copyright notices which must be updated. (Trac #3700, git xxx)".
BTW we need to run the first script daily, I'll discuss about details in the team list...

comment:7 Changed 4 years ago by tmark

  • Owner changed from UnAssigned to tmark

comment:8 follow-up: Changed 4 years ago by tmark

  • Owner changed from tmark to fdupont
  • Total Hours changed from 0 to 2

I ran the scripts and they all seem to do what they're intended to do.

It might be better if the two perl programs had the .pl extension.

It would also be helpful if they had some explanatory commentary.

At this point you have two scripts:

  1. newcopyrigths-git - a list of all the files based on the existing copyrights file, updated with those files changed this year
  1. newcopyrights-repo - a list of all the files in the repo and the most recent

year indicated in their copyright text

The difference between being a list of files whose copyright is stale.

Q: What is the rest of solution? How do the copyright years in the source files
actually get updated?

Q: Could you explain why one needs the copyrights file. It seems to represent a list of the files in the repo and what we think is their most recent copyright year. Can't you build the list of files that have been changed this year and whose copyright is out of date without it?

For each file changed this year:
   if it exists in the repo and it's copyright text is stale
         take some action 


    Where "take some action" could simpy be to output the filename or    
       actually update the copyright text in the file.

comment:9 in reply to: ↑ 8 ; follow-up: Changed 4 years ago by fdupont

Replying to tmark:

I ran the scripts and they all seem to do what they're intended to do.

=> fine!

It might be better if the two perl programs had the .pl extension.

=> they have no in bind9 by I agree they should.

It would also be helpful if they had some explanatory commentary.

=> you mean more comments inside the code, don't you?

At this point you have two scripts:

  1. newcopyrigths-git - a list of all the files based on the existing copyrights file, updated with those files changed this year
  1. newcopyrights-repo - a list of all the files in the repo and the most recent

year indicated in their copyright text

=> yes

The difference between being a list of files whose copyright is stale.

=> yes, you have 3 files which should match.

Q: What is the rest of solution? How do the copyright years in the source files
actually get updated?

=> manually (automatically update is complex, error prone and in fact even if bind9 has it is not done this way).

Q: Could you explain why one needs the copyrights file.

=> it is the current state maintained manually.

It seems to represent a list of the files in the repo and what we think is their most recent copyright year. Can't you build the list of files that have been changed this year and whose copyright is out of date without it?

=> yes but this doesn't catch all possible errors. In fact the copyrights file should be generated once (too painful to do it by hand) and after we should work on differences (smaller and errors/problems are visible).

For each file changed this year:
   if it exists in the repo and it's copyright text is stale
         take some action 

    Where "take some action" could simpy be to output the filename or    
       actually update the copyright text in the file.

=> I disagree after years of doing this for bind9: it is far better to have manual control with a tool showing errors/problems than to wait the fully smart stuff to become crazy (or to make us crazy :-).

Anyway I know the doc must be improved so where and how?

comment:10 in reply to: ↑ 9 ; follow-up: Changed 4 years ago by tmark

Replying to fdupont:

Replying to tmark:

I ran the scripts and they all seem to do what they're intended to do.

=> fine!

It might be better if the two perl programs had the .pl extension.

=> they have no in bind9 by I agree they should.

It would also be helpful if they had some explanatory commentary.

=> you mean more comments inside the code, don't you?

Yes, inside the code please.

At this point you have two scripts:

  1. newcopyrigths-git - a list of all the files based on the existing copyrights file, updated with those files changed this year
  1. newcopyrights-repo - a list of all the files in the repo and the most recent

year indicated in their copyright text

=> yes

The difference between being a list of files whose copyright is stale.

=> yes, you have 3 files which should match.

Q: What is the rest of solution? How do the copyright years in the source files
actually get updated?

=> manually (automatically update is complex, error prone and in fact even if bind9 has it is not done this way).

Q: Could you explain why one needs the copyrights file.

=> it is the current state maintained manually.

It seems to represent a list of the files in the repo and what we think is their most recent copyright year. Can't you build the list of files that have been changed this year and whose copyright is out of date without it?

=> yes but this doesn't catch all possible errors.

What sort of errors would get missed?

In fact the copyrights file should be generated once (too painful to do it by hand) and after we should work on differences (smaller and errors/problems are visible).

For each file changed this year:
   if it exists in the repo and it's copyright text is stale
         take some action 

    Where "take some action" could simpy be to output the filename or    
       actually update the copyright text in the file.

=> I disagree after years of doing this for bind9: it is far better to have manual control with a tool showing errors/problems than to wait the fully smart stuff to become crazy (or to make us crazy :-).

Hmm. I can't speak for Tomek but I think people's expectations for this ticket is that source file copyrigthts would be updated automatically. I'm not arguing that doing so could pose some challenges with differing file types and the changes would still have to be committed and so forth.

The system you have here is basically a check to alert us that copyrigths are stale. That being the case, I think at a minimum there should be a one-step process that a developer can run on a branch that outputs a list of copyright updates needed. That way a reviewer or a build script can simpy run "check_copyrights" and get $? = 0 if none are needed or $? > 0 and list of needed updates otherwise.

Q: How is this integrated into the Bind development process?

Anyway I know the doc must be improved so where and how?

I imagine this would need to go in the Kea Developer's guide.

comment:11 in reply to: ↑ 10 ; follow-up: Changed 4 years ago by fdupont

Replying to tmark:

=> yes but this doesn't catch all possible errors.

What sort of errors would get missed?

=> missing files (the problem is the copyrights file is used also as input).

Hmm. I can't speak for Tomek but I think people's expectations for this ticket is that source file copyrigthts would be updated automatically. I'm not arguing that doing so could pose some challenges with differing file types and the changes would still have to be committed and so forth.

=> fully automatic update is too dangerous for the small benefit: what we need is a way to check when copyrights come out of phase.

The system you have here is basically a check to alert us that copyrigths are stale. That being the case, I think at a minimum there should be a one-step process that a developer can run on a branch that outputs a list of copyright updates needed. That way a reviewer or a build script can simpy run "check_copyrights" and get $? = 0 if none are needed or $? > 0 and list of needed updates otherwise.

=> you assume copyrights are not correctly managed and are critical. I prefer to follow the way it is done in bind9: a daily script warns when a copyright year is wrong in a few branches.

Q: How is this integrated into the Bind development process?

=> daily script and manual update/

Anyway I know the doc must be improved so where and how?

=> BTW I added some comments in scripts.

I imagine this would need to go in the Kea Developer's guide.

=> I'll write a .dox file.

comment:12 Changed 4 years ago by fdupont

  • Owner changed from fdupont to tmark

Done, ready for another review.

comment:13 in reply to: ↑ 11 ; follow-up: Changed 4 years ago by jreed

Replying to fdupont:

=> fully automatic update is too dangerous for the small benefit: what we need is a way to check when copyrights come out of phase.

For several years I had simple system to report it for bind10 and then kea.
http://kea.isc.org/~tester/kea/LATEST_BUILD/old-copyrights
http://kea.isc.org/~tester/kea/LATEST_BUILD/missing-copyrights

=> you assume copyrights are not correctly managed and are critical. I prefer to follow the way it is done in bind9: a daily script warns when a copyright year is wrong in a few branches.

Q: How is this integrated into the Bind development process?

=> daily script and manual update/

bind9 has the check and update. The bind9 scripts can do the updates and even commit/push the changes. (See the bind9 git log for the tbox account doing "update copyright notice" and "newcopyrights".) If cannot handle it and email will be sent for warning.

comment:14 follow-up: Changed 4 years ago by tmark

  • Owner changed from tmark to fdupont

The intent of the ticket is to automatically update copyrights on files that have changed as doing it manually is "too tedious".

If the solution is not going to do automatic updates then I think it is overkill. It uses multiple scripts and a reference file (which must be updated) to generate a list of files that have stale copyrights.

I can get similar output from bash script something like this:

if [ ! -z $1 ]
    compare_with="$1"
then
    compare_with="HEAD"
fi

year=`date +%Y`
for name in  `git diff $1 --name-only | grep -v Makefile`
do
    str=`grep -m1  "Copyright" $name | grep -v $year`
    if [ ! -z "$str" ]
    then
        echo $name:$str
    else
        c=`grep -c "Copyright" $name`
        if [ $c -eq 0 ]
        then
            echo $name:Missing Copyright entry?
        fi
    fi
done

I believe the bind9 behavior Jeremy descibed above is what was expected as the outcome of this ticket.

Last edited 4 years ago by tmark (previous) (diff)

comment:15 in reply to: ↑ 13 Changed 4 years ago by fdupont

Replying to jreed:

Replying to fdupont:

=> fully automatic update is too dangerous for the small benefit: what we need is a way to check when copyrights come out of phase.

For several years I had simple system to report it for bind10 and then kea.
http://kea.isc.org/~tester/kea/LATEST_BUILD/old-copyrights
http://kea.isc.org/~tester/kea/LATEST_BUILD/missing-copyrights

=> why do you seem to be the only person to know this?

=> you assume copyrights are not correctly managed and are critical. I prefer to follow the way it is done in bind9: a daily script warns when a copyright year is wrong in a few branches.

Q: How is this integrated into the Bind development process?

=> daily script and manual update/

bind9 has the check and update.

=> yes but according to my experience I often have to update manually. Perhaps I am not lucky?

The bind9 scripts can do the updates and even commit/push the changes. (See the bind9 git log for the tbox account doing "update copyright notice" and "newcopyrights".) If cannot handle it and email will be sent for warning.

=> I received so many such emails I though it was the common case.

comment:16 in reply to: ↑ 14 Changed 4 years ago by fdupont

Replying to tmark:

The intent of the ticket is to automatically update copyrights on files that have changed as doing it manually is "too tedious".

=> tedious but safe.

If the solution is not going to do automatic updates then I think it is overkill. It uses multiple scripts and a reference file (which must be updated) to generate a list of files that have stale copyrights.

=> it is a check mechanism (vs a blind update one).

I can get similar output from bash script something like this:

if [ ! -z $1 ]
    compare_with="$1"
then
    compare_with="HEAD"
fi

year=`date +%Y`
for name in  `git diff $1 --name-only | grep -v Makefile`
do
    str=`grep -m1  "Copyright" $name | grep -v $year`
    if [ ! -z "$str" ]
    then
        echo $name:$str
    else
        c=`grep -c "Copyright" $name`
        if [ $c -eq 0 ]
        then
            echo $name:Missing Copyright entry?
        fi
    fi
done

=> this script is far to do the same. For instance it doesn't show COPYING is stale...

I believe the bind9 behavior Jeremy descibed above is what was expected as the outcome of this ticket.

=> giving up: IMHO there are more urgent things that to write and debug a blind update tool.

comment:17 Changed 4 years ago by fdupont

  • Owner changed from fdupont to UnAssigned

comment:18 Changed 4 years ago by fdupont

  • Resolution set to wontfix
  • Status changed from reviewing to closed

comment:19 Changed 4 years ago by fdupont

  • Resolution wontfix deleted
  • Status changed from closed to reopened

comment:20 Changed 4 years ago by fdupont

Reset status (no way to set it to new).

comment:21 Changed 3 years ago by tomek

  • Milestone changed from Kea1.1 to Outstanding Tasks

We're back in shoving tickets off from the current milestone. We decided on 2016-06-02 call to move this ticket to Outstanding.

Note: See TracTickets for help on using tickets.