Your Best Friend
Do you like us? Become our BFF by being one of the first 1000 users to donate $10,- or more!
And as the name implies, the BFF-status will be forever. Unless you sleep with my girlfriend, our friendship will be over then.
22 March 2018
I screwed up gents. While syncing to new storage and having seen complete folders on the destination, I wiped a few folders on the source to make some room. A filesystem error occurred on the destination and somehow everything got wiped on the destination without even a fsck being ran. The folders were just gone. 2.6m NZBs got lost. 12m still remaining.
Five years of work and over 10% just went poof. I don't feel defeated. I'll make sure y'all get these releases back one way or the another. I have been saving up NZBs for the last year or so, these I will import today.
I've been trying to get more people to chat as I'm very active there. I'm always on the desktop client and have been messaged about a problem sometimes occurring on web clients, disconnecting frequently. Have taken another good look at chat and would like to invite y'all to try it again. Peace, out!
9 March 2018
Lots of fixes today, please keep reporting them to me on chat or Issues. I doubled all BFF limits too. For fixed issues see http://gitlab.presse-papier.fr/OptimusPrime/usenet-crawler/issues?state=closed Peace, out!
8 March 2018
Hello friends, sure had a rough start this year. For months I have been having power issues. I couldn't get to terms with the landlord which resulted in a civil suit. I got out clean but I've been feeling pretty lost losing my house. I'm currently squatting getting things together and finally at a stage allowing me to work a fair bit every day on the sites. Some people are really angry and I understand but this is a one man show. Megatron nr2 left pretty quickly and apart from the work involved there is also the mental ability needed to handle all the pushback involved in certain media.
When the madness happened I increased everyone limit's to accommodate their clients. The 99% who got their cookie got lost in the 1% who felt like they lost their home. I want to assure you I'm not going anywhere so please stick around and give me another chance, we are best friends for a reason <3
Also everyone who wants to vent or just talk or needs help, chat is going to be the place to be. Find me at http://chat.presse-papier.fr/direct/OptimusPrime
25 January 2018
We survived a filesystem corruption on our assets drive.
404 on NZB's should be resolved asap as we are merging backup's. This happened due to a writebacked cached lvm volume crashing. Will not use that again...
Some good news. I have enabled aggressive caching for smooth browsing and also am in the process of rolling out a new codebase. We have been hacking changes in quite a lot and want to separate these. Some custom option might break while we roll this out in the next few weeks. Progress - Direct chat to admin Peace, out!
18 January 2018
Giving back is good. As promised the spotweb api details.
API URL: http://spotweb.presse-papier.fr/newznab
API KEY: f66c2db74696361a28c757c8a26f0c12
Enjoy my friends!
13 January 2018
Got quite a few messages about the new chat system. All negative ;-) Jeez, stop being such a girl. Daddy has got you something though, you may remember the good times. THE OLD CHAT IS BACK. GOGO PS user integration will come later, for now please register a nick using your username.
4 January 2018
Like most of you I don't like the new chat much. Reason we switched is that writing a single sign one for this one was quite easy. I would like to go back to the old chat and am looking for your assistance. Are you familiar with mongodb and php and THIS DOES NOT SCARE YOU please contact us and we will compensate you for your time. Peace, out!
3 January 2018
Another indexer has been added to your menu called Spotweb. NZB's found there are different from UC, we have thought about merging but feel this is breaking an unspoken rule. There is an API for it too, will release details later. Peace, out!
2 January 2018
Too Many Requests...FIXED?
Do you also hate to see that annoying message? Well we have a solution for you: API reset button!
You can find this fantastic solution in your profile EDIT page
Have an awesome day!
27 December 2017
What's up, they call me [email protected] I think I am the luckiest person on this planet, because I've a loving family, good friends, and a beautiful platform called 'Usenet Crawler', which allows me to spend most of the time by doing geeky stuff. Thank you all that I can be a part of this great experience. See you on the forums and chat.
20 September 2017
Hey hey, I took the NFO search down. This beast was responsible for about 40 GB of RAM. Should make the site more snappy. There are quite a few outstanding issues but due to moving I won't be able to finish those soon I'm afraid. Peace, Out!
29 August 2017
Been looking at ways to save money. Cancelled one license we could do without and have been monitoring RAM usage closely. I think we can run again on 128 GB RAM easily with the NFO index.
I'll be monitoring a few days more to be sure. If we push through down time should not be more than 10 minutes and will give a tremendous saving. Those scrapers really made things much harder :-( I got many messages about the new limits being low for our regulars. I agree but it's not so easy to solve. I'm trying to go forward with a button in your profile which will add more calls. I'm not sure yet though how to stop a proper scraper from doing the same. I don't think it's possible. If you have an idea please bring it to me. Peace, out!
28 August 2017
I've had a pretty rough week in the sense I'm not sure how to proceed. Basically this happened in last year.
It all started with a theft of all my BTC from UC which was about one year of savings. Saved up on an exchange with 2fa email because back in those days I would trade daily short or long and discuss with a fellow co-worker. We had a lot of fun. One day I got fired for some stupid argument anyone can shake on, but they went on work computer and stole all pretty much immediately. Cops were useless and violence only seemed to result in more bullshit.
I intended to never release this to you guys, cos we have had our fair share of sad stories about other websites going poof and people tend to discredit my life. Anway months with vivid dreams with me ending in rehab and sending out the very first request ever for help. You guys helped me so much through it, I ignored the haters and read all the wonderful compliments and wishes.Without you usenet crawler would not exist, simply cos I wouldn't run it.
Here and there I read some "discussions" about UC. What does it cost and how much power do you need. Since I'm banned from reddit (when will you guys ever give up on being dicks) I thought I would share them here as it's interesting data after so many years.
As we know UC has been scraped a lot. I will give you the old data.
- Maximum TB ever per month used: 50 TB
- Average between 15 and 20 at the moment.
- We have had times when we served pages at 25 MB/s
- Max hits per hour: 500k
- Avg hits per hour: 250-300k
- 11.610.116 releases indexed
- 4 TB web files
- 300 GB mysql database
- About 2% donates willingly (before sorry :-(
So yeah thats thousands of people requesting a lot of narrow data in a huge table. Some proper russians made sphinxsearch enabling us basically to trade CPU for RAM which makes a tremendous difference.
Like two years ago we paid about $400 a month for dual 2620s with big disks and ssd's and 128 GB RAM and that ran fine but that company got into a bit of trouble. So I changed it a bit around hoping we would save some money in a year or so by switching providers.
- One main having dual 26x0s 256 GB RAM two sata's two ssd's, that costs is about 450 a month
- One backup having single 12xx 256 GB RAM two sata two ssd's, that costs us about 250 a month
- In front we use $50 proxy a month.
Few more things left, I need 4 usenet subs and have to keep staff happy for about $150 a month and about 80 for software licenses.
I'm thinking to remove some RAM and remove NFO searching as it's hardly used and takes a good 60 GB of RAM. See if we can hold with 128. Should be easy. Question remains do we put away server 1 or 2 or keep em both? Leave your feedback on chat.
On a side note we fixed a bug in everyone profile, the user downloads shown was totally messed up due to an IP swap. That's has been resolved so keep em coming on Issues. To support the site better as we are feeling the long term BFF's bit there, in the near future there will be a few new exciting ads. As always good stuff and non instrusive ;-) Peace, out!
21 August 2017
Just now a fellow pointed out that our ridiculous limits are unnecessary. My intention is to offer the best possible experience and having low limits is something I frowned upon. Oh boy, was I wrong. I've done some basic analysis on our downloads and api requests and it knocked my socks right off...
BFF
nr 1: 46544 & 5001
nr 10: 12028 & 475
nr 100: 3932 & 56
lowest from top 1%: 2765 & 31
Regular:
nr 1: 10272 & 1002
nr 10: 10006 & 951
nr 100: 6334 & 160
lowest from top 1%: 149 & 4
These results can only mean two things. Regular users are way less intelligent and use bad clients or we have an army of scrapers. With the current bills and return I'm not so sure if we can keep up for another year. So after much thought I'm sorry to say but I'm going to break one of my rules and lower the limits tremendously. We are only hurting usenet and the site if we keep going on like this. New limits: Regular 250 & 25, BFF 5000 & 500. If you need more, you can hit me up for a manual increase. Peace, out!
18 August 2017
It's me again. So I've been wondering why no one posted issues on our issue tracker and to my surprise the link isn't working in the menu. Stupid huh. Anyway, you can bug me now using the link in menu called Issues. Please keep them to the site only. Real life issues can be talked about in chat, we have professionals standing by. Also our proxy sucks, I rolled it back two versions last night and it crashes way less. In case anyone has experience with openlitespeed, hit me up. I'll probably ditch it soon for nginx like we always had. Peace, out!
17 August 2017
503s, certificate issues, man, it's like usenet-crawler doesn't know what the hell he's doing sometimes. "Computer says no". After fixing the issues friends came back together to improve UC even more. I'm not even going to tell you the changes as they are pretty obvious. Oh yeah, our email address has become unusable. I'm getting spammed by the 100s every day for months now, I've been able to filter out 90% but it still taking lots of time to go through it all. It;s better to reach out in chat currently. Peace, out!
8 August 2017
I would like to welcome a new mod Blitzwing. His main priority will be dealing with chat and payment related matters. Close buddy so no bullshit. Usenet is expected to pick up in the next year. I have learned quite a few new traits as the more closely observing friends have noticed. 4 years later and I still so many possibilities. It will be great for sure but also I expect technology to change a little bit. We are going to see more full service solutions popping up, it's not about the nzb's anymore. I have decided it's time for a forum, a proper one. While I'm very fond of our lair, our princess has to come out once. I want my friends to step up and reclaim our kingdom. I have already instructed all weapons to be sharpened. After we are done, we can all hang out in our living room.
Blitzwing has resurrected our issue tracker. Want to shame the mods on bugs that havent 't been fixed? Found a exploit and want your bounty that's the place to get professional help. You can find the links in your menu. Peace, out!
11 July 2017
Since my last update we have had trouble indexing scene releases, this has been resolved. Enjoy ;-)
14 June 2017
We currently have a lot of spare room on our cluster. Are you looking currently for a vpn, vps or a dedicated server make sure to look OptimusPrime in chat for a nice deal. Peace, out!
13 June 2017
We have been running real smooth. Plenty of resources left to at least triple our users ;-) Everything is pretty much back as it was. I'll bring the issue tracker online asap so we can finish the last bits together.
There is a lot of news about ether so I've added the option to donate using ether coins. Peace, out!
1 June 2017
Chat has been restored: Chat
Spotweb has been restored: Spotweb login: ucrawler pass: usenet
30 May 2017
currently adding missing uc files
nfo searching fixed (many nfo's missing)
fixed 503 error (false alarms got triggered)
api search fixed (problem in cdn)
28 May 2017
Hello friends, Best Friends Forever and those who slept with my girlfriend.
We're back!
Our last migration did not solve our problems, this forced me to radically change the set-up. We are no longer virtualizing anything!
Not all of our services are up and running completely yet, here's a quick rundown of their statuses
UC RUNNING 95%
Nzedb OFFLINE
Spotweb OFFLINE
Chat OFFLINE
Issue Tracker OFFLINE
I'm currently working hard on the offline services, they will be up and running ASAP.
We owe all of you our most sincere apologies, we did not anticipate a downtime of this duration.
From 2nd june to 4th june everyone will be BFF, let's call it TBF (Temporary Best Friend).
Live Chat Support will be available for BFF's only. We are very grateful for their support and they therefor deserve the best service we can offer.
So if you got any questions please hop on the chat and we will support you the best we can.
Enjoy your stay
19 Feb. 2017 = Welcome back again
I ran into a few issues migrating the servers to one. API open connections are looking really good now, 200 compared to 5000.
On top I got jumped for the first time ever today, bunch of scoundrels. I'm not too hurt but had a long day and joints are hurting. Please bear with the last bugs and tomorrow it's party time!
5 Feb. 2017
Hey y'all. We had a boot disk failure this week on one of our cluster machines. Due to miscommunication with data center restoring the hardware took longer as expected. I probably should have migrated a few services but I didn't expect this to happen, usually we are spot on with replacing failed hardware.
We took it a bit easy this week with another failing machine but have made a change so releases are processed way sooner. Should help out whack out those bad releases as quick as they get in.
With all the changes as of late I feel my debt has been re payed to you guys. We have a few staff now, all bugs since neglecting the site have been squashed and we have a lot in store coming soon.
One of the new things we'll be pushing this week is our BFF reseller program. In a nutshell everyone with access to BTC can buy bulk BFF from us. These bulk accounts can then be resold for $13 instead of the regular $10. Our aim is to allow anyone to become a reseller. Buyers can select their seller based on past transaction history. The chat will have a room dedicated for it. I have my hopes set up for this. I believe if we make this happen we won't need to not change our limits, ads or whatever ever. We are still doing good, so no need for alarm, but I'm expecting growth this year. Let me know what you think @OptimusPrime on chat. Peace, out.
25 Jan. 2017
Doubt thou the starts are ire;
Doubt that the sun doth move;
Doubt thruth to be a liar;
But never doubt I love.
In which TV show can we find this poem?
First three to hit up @OptimusPrime gets to become a BFF
We have been stuck for almost two months with a half working API since we moved servers.
Well it's finally solved now! So read the poem again and never have doubt anymore. From now its up up up!
Also all search parmeters working again, you can filter one again on subs lang and/or audio lang!
Jan 18, 2017
I've kept y'all waiting long enough. Here is my next gift for all my friends! Incoming releases have now at least doubled in numbers. Sharing is caring. You better not forget we have two more indexers totally free. Just get your bottom into chat and ask around. Peace, out.
Jan 14, 2017
While removing a node from our cluster I made a mistake causing split brain and loss of configuration files for all our servers. Worked through the night, manually fixing all from memory. We have changed our backup procedures as well to avoid such a terrible onslaught from happening again. Have a good one!
Jan 13, 2017
Due to an upgrade problem of the chat server software which can not be rolled back at the time chat functionality will be temporarily down, sorry for the hassle. On the good news front however we are glad to announce that we are still working hard on improving the overall performance of the site and increasing the support for various API clients. Our TV API is working once again for everyone, yay! Very happy we could fix it instead of resorting to drastic measures. We have only tested Sonarr at the moment. Please hit a mod up in chat if you have trouble with other clients. http://chat.presse-papier.fr is where you should be, we will be giving away many goodies in a few days.Dec 26, 2016