Current issues

lukas_demetz

Current issues (updated whenever needed):

Namibia Systems:

Upload speed greatly degraded due to a routing issue of Telecom Namibia. It is being worked on (open since 5 march 2021). Seemingly improved since 19th may.

iDK 20:

The new camera brought a great improvement in image quality, however we do experience some odd, sporadic horizontal banding that seems to be caused by some sort of interference. We are working with FLI to resolve this issue.
Update: A workaround for the horizontal banding has been found: Simply use 1 MHz readout mode on the iDK if you are afraid of those (mainly for photometry and pretty pictures, I'd say).

Veloce Namibia:

Greatly improved field flatness and star shapes, to the point that most bin 2x2 images are fully usable. Bin 1x1 is still tricky as there is some sag in the imaging train caused by the way the camera is connected to the train three screws, go figure...)
~~Some frame sare downloaded with an odd checkered composition. Seems a driver issue.~~ Solved We had to change a driver setting, seems reliable now.
Reflection rings have been mostly eliminated with a longer light shield extension 😀

Veloce Spain:

Currently using "old" computer. The new one was experiencing sporadic reboots without obvious issues. The computer is currently on the way back to the observatory, expecting it to be put back in use around 14th april 2021.
A consequence of using the old computer: Fixed time plans do not start, and generally the whole system is incredibly slow
18th may 2021: New computer installed and working nominally now

lukas_demetz

Seemingly in the wake of the connection issues we lost some frames during the upload. Please check and let me know, we will refund them of course. Still trying to figure out what exactly happened 🤐

lukas_demetz

Regarding the lost data that we seemingly had over the last weeks, finally we were able to resolve it:
In short, those "lost" frames ended up on the "old" Google Drive instead of the new storage 😅

I copied it now to the new system, and you should see it shortly there. I am relieved, considering it's about 55GB (!) of missing data.

As for the internet connection in namibia, it is currently basically unusable. Working on it, but it is more tedious than I hoped.

lukas_demetz

While we wait for Telecom Namibia to get things fixed, we were able to at least establish a slow link to upload some data.

To speed things up a bit, we are now compressing the frames with 7Zip. You might need that (free) application to decompress the frames. We are saving about 55% of space by compressing them 😀

lukas_demetz

The connection issues in Namibia are more difficult to get sorted than expected. I will upload all data we got until 14th may morning today. Everything taken afterwards will be affected again by those slow, spotty connections.

What I found out is that they need to replace a particular microwave backhaul hardware from Ceragon, however they seem to struggle to find a replacement unit. Luckily for us, another big scientific project (lookup for the H.E.S.S.) is affected by the same problem, so they are pushing hard for this issue being solved.

I will keep you updated. Worst case we will move the data with usb sticks between europe and namibia every few weeks...

lukas_demetz

Seems that since today the connection is more stable. Not fast, far from it, but it seems to not drop out every minute or so.

This means first of all that the submission and updates to and from the observatory will be less error-prone. And also that we will be able to investigate issues more easily.
I will keep an eye on the available upload speed. If it improves, I will switch back to upload the frames without compression 😀

lukas_demetz

Update: The compression and splitting is now disabled. Last night's frames were already all uploaded, so it seems that we are getting back to normal operations 😃

lukas_demetz

Update: The Veloce in Namibia has now a new pointing model. I noticed that tracking was degraded, but should now be good again.

lukas_demetz

Update: Finally the new, improved calibration masters have been loaded onto the Veloce compute rin Namibia. The automatic calibration should now work correctly again.

lukas_demetz

The connection issues in Namibia have finally been resolved. Look at this:

The uploads are again direct to the fileserver, which means that usually the finished frames will be uploaded within minutes from being taken 😀